[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:17 dist-test-slave-chsk krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:17 dist-test-slave-chsk krb5kdc[21684](info): set up 2 sockets
Aug 10 14:35:17 dist-test-slave-chsk krb5kdc[21684](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 14:35:18.938891 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:19.079789 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:46031
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:46031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:19.200579 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:19.200953 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:19.201040 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:19.205367 21698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:19.205487 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:19.205529 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:19.205570 21698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:19.205603 21698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:19.211429 21698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:46031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:46031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:19.212741 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:19.213943 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:19.224529 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:19.224620 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:19.224531 21698 server_base.cc:1034] running on GCE node
W20260810 14:35:19.224784 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:19.225602 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:19.226995 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:19.228302 21698 hybrid_clock.cc:648] HybridClock initialized: now 1786372519228253 us; error 98 us; skew 500 ppm
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:19.235435 21698 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:19.236696 21698 webserver.cc:460] Webserver started at http://127.21.43.126:41113/ using document root <none> and password file <none>
I20260810 14:35:19.237550 21698 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:19.237670 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:19.237938 21698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:19.240321 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "56abab5c8e6a4989bc8a9e678a3917fd"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "f1d13bbb232d5e6d43f7db69da315f42"
server_key_iv: "c1ce37f5c8199a16b42f9e1a56c58293"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.240914 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "56abab5c8e6a4989bc8a9e678a3917fd"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "f1d13bbb232d5e6d43f7db69da315f42"
server_key_iv: "c1ce37f5c8199a16b42f9e1a56c58293"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.245316 21698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 14:35:19.248373 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.249784 21698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:19.249962 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "56abab5c8e6a4989bc8a9e678a3917fd"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "f1d13bbb232d5e6d43f7db69da315f42"
server_key_iv: "c1ce37f5c8199a16b42f9e1a56c58293"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.250093 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:19.261765 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:19.265245 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:19.265487 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:19.273365 21698 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:46031
I20260810 14:35:19.273409 21773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:46031 every 8 connection(s)
I20260810 14:35:19.274416 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:19.277521 21774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:19.278623 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 21698
I20260810 14:35:19.278798 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:19.280591 21677 external_mini_cluster.cc:1428] Setting key dbfb11910907744769ddf143f01b7568
I20260810 14:35:19.285521 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd: Bootstrap starting.
I20260810 14:35:19.288285 21774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:19.289095 21774 log.cc:824] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:19.291630 21774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd: No bootstrap required, opened a new log
I20260810 14:35:19.296409 21774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } }
I20260810 14:35:19.296726 21774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:19.296777 21774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56abab5c8e6a4989bc8a9e678a3917fd, State: Initialized, Role: FOLLOWER
I20260810 14:35:19.297308 21774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } }
I20260810 14:35:19.297452 21774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:19.297544 21774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:19.297685 21774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:19.299161 21774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } }
I20260810 14:35:19.299587 21774 leader_election.cc:302] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56abab5c8e6a4989bc8a9e678a3917fd; no voters:
I20260810 14:35:19.299952 21774 leader_election.cc:288] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:19.300114 21779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:19.300371 21779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [term 1 LEADER]: Becoming Leader. State: Replica: 56abab5c8e6a4989bc8a9e678a3917fd, State: Running, Role: LEADER
I20260810 14:35:19.300819 21779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } }
I20260810 14:35:19.301528 21774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:19.302194 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } } }
I20260810 14:35:19.302350 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [sys.catalog]: This master's current role is: LEADER
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:19.302760 21785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:19.304702 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56abab5c8e6a4989bc8a9e678a3917fd. Latest consensus state: current_term: 1 leader_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56abab5c8e6a4989bc8a9e678a3917fd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 46031 } } }
I20260810 14:35:19.304828 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:19.307305 21785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:19.310551 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:19.284752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.21.43.126:46031)
0810 14:35:19.285318 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:19.285338 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:19.286457 (+ 1119us) server_negotiation.cc:407] Connection header received
0810 14:35:19.290499 (+ 4042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:19.290518 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:19.290853 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:19.291200 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:19.294389 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.295481 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.297206 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.297907 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.298079 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:19.303819 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:19.303827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:19.303843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:19.303884 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:19.306245 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.306849 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.306856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.306861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.307009 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.307488 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.307492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.307493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.308068 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:19.308314 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:19.309073 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 14:35:19.309395 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":163,"threads_started":1}
I20260810 14:35:19.315037 21785 catalog_manager.cc:1347] Generated new cluster ID: d7ce7191832f444c8dafcf69814de966
I20260810 14:35:19.315158 21785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:19.340822 21785 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:19.342228 21785 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:19.352253 21785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56abab5c8e6a4989bc8a9e678a3917fd: Generated new TSK 0
I20260810 14:35:19.353212 21785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:19.388792 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:19.513818 21802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:19.514138 21802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:19.514187 21802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:19.519023 21802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:19.519122 21802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:19.519250 21802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:19.525331 21802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:19.526579 21802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:19.527766 21802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:19.539577 21811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:19.539732 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:19.540019 21802 server_base.cc:1034] running on GCE node
W20260810 14:35:19.539573 21808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:19.540570 21802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:19.541326 21802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:19.542615 21802 hybrid_clock.cc:648] HybridClock initialized: now 1786372519542594 us; error 39 us; skew 500 ppm
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:19.545989 21802 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:19.547214 21802 webserver.cc:460] Webserver started at http://127.21.43.65:37687/ using document root <none> and password file <none>
I20260810 14:35:19.547849 21802 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:19.547947 21802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:19.548221 21802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:19.550504 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "385682e6264c4c35859beb8df7587954"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "44f66b2e8384b41a4e9f61e9a4d7744e"
server_key_iv: "6d7dbc68f87cbae5da4eb5b27aade781"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.551290 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "385682e6264c4c35859beb8df7587954"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "44f66b2e8384b41a4e9f61e9a4d7744e"
server_key_iv: "6d7dbc68f87cbae5da4eb5b27aade781"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.555630 21802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:19.558514 21820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.560060 21802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:19.560232 21802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "385682e6264c4c35859beb8df7587954"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "44f66b2e8384b41a4e9f61e9a4d7744e"
server_key_iv: "6d7dbc68f87cbae5da4eb5b27aade781"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.560376 21802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:19.575331 21802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:19.579686 21802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:19.579932 21802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:19.580621 21802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:19.581543 21802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:19.581616 21802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.581686 21802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:19.581735 21802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.591409 21802 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:41435
I20260810 14:35:19.591452 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:41435 every 8 connection(s)
I20260810 14:35:19.592382 21802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:19.597399 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 21802
I20260810 14:35:19.597532 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:19.597826 21677 external_mini_cluster.cc:1428] Setting key 6edc4104a9ae9e3064b54bc38efd5e64
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:19.609504 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:19.594520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:34395 (local address 127.21.43.126:46031)
0810 14:35:19.594700 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:19.594706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:19.595646 (+ 940us) server_negotiation.cc:407] Connection header received
0810 14:35:19.596939 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:19.596942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:19.596993 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:19.597072 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:19.599103 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.599723 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.600997 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.601246 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.601364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:19.605257 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:19.605261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:19.605263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:19.605294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:19.606994 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.607774 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.607779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.607781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.607873 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.608282 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.608287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.608289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.608504 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:19.608628 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:19.609181 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 14:35:19.609330 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:35:19.610436 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:19.594914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:46031 (local address 127.21.43.65:34395)
0810 14:35:19.595479 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:19.595526 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:19.596733 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:19.597361 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:19.597372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:19.597916 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:19.598917 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:19.598933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.599877 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.599880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:19.600832 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:19.600843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.601423 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.601427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:19.601722 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:19.602954 (+ 1232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:19.602984 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:19.604993 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:19.607252 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:19.607261 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:19.607284 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:19.607636 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:19.607982 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:19.607985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:19.607987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:19.608169 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:19.608620 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:19.608627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:19.608915 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:35:19.609139 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 14:35:19.609514 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":180,"threads_started":1}
I20260810 14:35:19.612005 21944 heartbeater.cc:347] Connected to a master server at 127.21.43.126:46031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:19.612303 21944 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:19.612950 21944 heartbeater.cc:511] Master 127.21.43.126:46031 requested a full tablet report, sending...
I20260810 14:35:19.615042 21732 ts_manager.cc:194] Registered new tserver with Master: 385682e6264c4c35859beb8df7587954 (127.21.43.65:41435)
I20260810 14:35:19.616756 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:34395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:19.635653 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:19.766855 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:19.767257 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:19.767315 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:19.772101 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:19.772192 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:19.772259 21949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:19.778254 21949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:19.779491 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:19.780548 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:19.790997 21956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:19.791113 21957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:19.791005 21959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:19.791306 21949 server_base.cc:1034] running on GCE node
I20260810 14:35:19.791833 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:19.792542 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:19.793773 21949 hybrid_clock.cc:648] HybridClock initialized: now 1786372519793758 us; error 35 us; skew 500 ppm
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:19.796864 21949 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:19.797883 21949 webserver.cc:460] Webserver started at http://127.21.43.66:37099/ using document root <none> and password file <none>
I20260810 14:35:19.798493 21949 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:19.798563 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:19.798753 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:19.801103 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "56e08ec03fdd4bd4b23aa5e945b9b2d1"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "b0c68b5e3c9b455b1e7ca7f75a49053b"
server_key_iv: "4c5b351491ed7884bc8c180d07fb4432"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.801761 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "56e08ec03fdd4bd4b23aa5e945b9b2d1"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "b0c68b5e3c9b455b1e7ca7f75a49053b"
server_key_iv: "4c5b351491ed7884bc8c180d07fb4432"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.806336 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:19.809232 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.810914 21949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:19.811066 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "56e08ec03fdd4bd4b23aa5e945b9b2d1"
format_stamp: "Formatted at 2026-08-10 14:35:19 on dist-test-slave-chsk"
server_key: "b0c68b5e3c9b455b1e7ca7f75a49053b"
server_key_iv: "4c5b351491ed7884bc8c180d07fb4432"
server_key_version: "encryptionkey@0"
I20260810 14:35:19.811278 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:19.838073 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:19.841770 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:19.842023 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:19.842736 21949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:19.843952 21949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:19.844043 21949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.844125 21949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:19.844177 21949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:19.854045 21949 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:37787
I20260810 14:35:19.854110 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:37787 every 8 connection(s)
I20260810 14:35:19.855082 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:19.855436 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 21949
I20260810 14:35:19.855553 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:19.855854 21677 external_mini_cluster.cc:1428] Setting key 9aeca17416b16f7134568ddd70632f11
Aug 10 14:35:19 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372519, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:19.871481 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:19.857388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:47111 (local address 127.21.43.126:46031)
0810 14:35:19.857542 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:19.857547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:19.858676 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 14:35:19.860122 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:19.860128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:19.860195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:19.860310 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:19.861641 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.862580 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.863972 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.864405 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.864543 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:19.867517 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:19.867523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:19.867529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:19.867568 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:19.868905 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.869498 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.869501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.869502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.869580 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:19.869994 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:19.869999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:19.870003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:19.870293 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:19.870410 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:19.871233 (+ 823us) server_negotiation.cc:299] Negotiation successful
0810 14:35:19.871353 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:35:19.872709 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:19.857930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:46031 (local address 127.21.43.66:47111)
0810 14:35:19.858491 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:19.858546 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:19.859904 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:19.860524 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:19.860532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:19.860940 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:19.861463 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:19.861477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.862783 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.862790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:19.863832 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:19.863843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:19.864608 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:19.864611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:19.864762 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:19.865440 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:19.865457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:19.867319 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:19.869020 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:19.869025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:19.869041 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:19.869383 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:19.869678 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:19.869680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:19.869681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:19.869881 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:19.870471 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:19.870478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:19.870867 (+ 389us) client_negotiation.cc:769] Sending connection context
0810 14:35:19.871128 (+ 261us) client_negotiation.cc:240] Negotiation successful
0810 14:35:19.871566 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":162,"threads_started":1}
I20260810 14:35:19.873978 22092 heartbeater.cc:347] Connected to a master server at 127.21.43.126:46031
I20260810 14:35:19.874251 22092 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:19.874908 22092 heartbeater.cc:511] Master 127.21.43.126:46031 requested a full tablet report, sending...
I20260810 14:35:19.876210 21732 ts_manager.cc:194] Registered new tserver with Master: 56e08ec03fdd4bd4b23aa5e945b9b2d1 (127.21.43.66:37787)
I20260810 14:35:19.876811 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:47111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:19.890507 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:20.006803 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:20.007109 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:20.007249 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:20.011462 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:20.011533 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:20.011631 22098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:20.017289 22098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:46031
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:20.018443 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:20.019560 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:20.029294 22105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:20.029330 22104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:20.029299 22108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:20.029657 22098 server_base.cc:1034] running on GCE node
I20260810 14:35:20.030036 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:20.030648 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:20.031805 22098 hybrid_clock.cc:648] HybridClock initialized: now 1786372520031779 us; error 37 us; skew 500 ppm
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:20.034487 22098 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:20.035413 22098 webserver.cc:460] Webserver started at http://127.21.43.67:44173/ using document root <none> and password file <none>
I20260810 14:35:20.035923 22098 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:20.035979 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:20.036150 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:20.038152 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "21f68dbe4c634c7081c4b26865083773"
format_stamp: "Formatted at 2026-08-10 14:35:20 on dist-test-slave-chsk"
server_key: "30eee81ade7991fd3d11926e63f95aa4"
server_key_iv: "9fefdb2a297605ee8ae34d35e451fcba"
server_key_version: "encryptionkey@0"
I20260810 14:35:20.038676 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "21f68dbe4c634c7081c4b26865083773"
format_stamp: "Formatted at 2026-08-10 14:35:20 on dist-test-slave-chsk"
server_key: "30eee81ade7991fd3d11926e63f95aa4"
server_key_iv: "9fefdb2a297605ee8ae34d35e451fcba"
server_key_version: "encryptionkey@0"
I20260810 14:35:20.042434 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:20.045015 22115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:20.046267 22098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:20.046427 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "21f68dbe4c634c7081c4b26865083773"
format_stamp: "Formatted at 2026-08-10 14:35:20 on dist-test-slave-chsk"
server_key: "30eee81ade7991fd3d11926e63f95aa4"
server_key_iv: "9fefdb2a297605ee8ae34d35e451fcba"
server_key_version: "encryptionkey@0"
I20260810 14:35:20.046566 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:20.066156 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:20.069515 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:20.069738 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:20.070452 22098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:20.071384 22098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:20.071455 22098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:20.071559 22098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:20.071612 22098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:20.081413 22098 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:37083
I20260810 14:35:20.081467 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:37083 every 8 connection(s)
I20260810 14:35:20.082425 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:20.088425 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22098
I20260810 14:35:20.088534 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:20.088815 21677 external_mini_cluster.cc:1428] Setting key 1ac4c230f453bbd7173bb84449d3708e
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:20.097296 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:20.084450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:58581 (local address 127.21.43.126:46031)
0810 14:35:20.084620 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:20.084623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:20.085471 (+ 848us) server_negotiation.cc:407] Connection header received
0810 14:35:20.086806 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:20.086810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:20.086873 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:20.086983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:20.088271 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.088882 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.090265 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.090572 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.090687 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:20.093607 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:20.093613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:20.093616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:20.093646 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:20.095096 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.095722 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.095725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.095727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.095812 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.096168 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.096170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.096171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.096376 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:20.096457 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:20.097030 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 14:35:20.097148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:35:20.098234 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:20.084817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:46031 (local address 127.21.43.67:58581)
0810 14:35:20.085315 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:20.085360 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:20.086582 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:20.087234 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:20.087242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:20.087624 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:20.088117 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:20.088130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.089022 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.089025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:20.090136 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:20.090146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.090732 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.090737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:20.090909 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:20.091784 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:20.091805 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:20.093367 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:20.095308 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:20.095314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:20.095327 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:20.095600 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:20.095906 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:20.095908 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:20.095909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:20.096058 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:20.096468 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:20.096473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:20.096796 (+ 323us) client_negotiation.cc:769] Sending connection context
0810 14:35:20.096972 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 14:35:20.097269 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260810 14:35:20.099507 22241 heartbeater.cc:347] Connected to a master server at 127.21.43.126:46031
I20260810 14:35:20.099793 22241 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:20.100350 22241 heartbeater.cc:511] Master 127.21.43.126:46031 requested a full tablet report, sending...
I20260810 14:35:20.101500 21732 ts_manager.cc:194] Registered new tserver with Master: 21f68dbe4c634c7081c4b26865083773 (127.21.43.67:37083)
I20260810 14:35:20.102301 21732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:58581
I20260810 14:35:20.103885 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:20.133581 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:20.124014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.21.43.65:41435)
0810 14:35:20.124297 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:20.124307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:20.124339 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:35:20.124391 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:20.124396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:20.124590 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:20.124713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:20.125387 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.126133 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.126954 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.127443 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.127550 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:20.130125 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:20.130136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:20.130150 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:20.130178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:20.131987 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.132400 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.132405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.132406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.132487 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.132727 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.132729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.132730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.132977 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:20.133143 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:20.133300 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 14:35:20.133427 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260810 14:35:20.134691 21852 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58386
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[21684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:20.165791 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:20.156806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.21.43.65:41435)
0810 14:35:20.156987 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:20.156991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:20.157170 (+ 179us) server_negotiation.cc:407] Connection header received
0810 14:35:20.157286 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:20.157289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:20.157334 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:20.157415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:20.158129 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.158653 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.159580 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:20.159821 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:20.159921 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:20.162517 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:20.162521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:20.162523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:20.162549 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:20.164057 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.164481 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.164483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.164485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.164559 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:20.164813 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:20.164815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:20.164816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:20.165030 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:20.165147 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:20.165462 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 14:35:20.165572 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:35:20.166913 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 21802
I20260810 14:35:20.176126 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 21949
I20260810 14:35:20.183732 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22098
I20260810 14:35:20.194751 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 21698
2026-08-10T14:35:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2412 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[22257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[22257](info): set up 2 sockets
Aug 10 14:35:20 dist-test-slave-chsk krb5kdc[22257](info): commencing operation
krb5kdc: starting...
W20260810 14:35:21.303764 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:21 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:21.451253 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40407
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:21.571748 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:21.572091 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:21.572178 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:21.576320 22271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:21.576416 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:21.576454 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:21.576515 22271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:21.576556 22271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:21.582113 22271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:21.583411 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:21.584525 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:21.593833 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:21.593829 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:21.594045 22271 server_base.cc:1034] running on GCE node
W20260810 14:35:21.593832 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:21.594875 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:21.596181 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:21.597349 22271 hybrid_clock.cc:648] HybridClock initialized: now 1786372521597330 us; error 34 us; skew 500 ppm
Aug 10 14:35:21 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:21.600665 22271 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:21.601783 22271 webserver.cc:460] Webserver started at http://127.21.43.126:40319/ using document root <none> and password file <none>
I20260810 14:35:21.602460 22271 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:21.602535 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:21.602725 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:21.604943 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "36201a0278ff4d648f657182295459c6"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "3d30b08170923dd437fbf6e0ce2bb219"
server_key_iv: "7aff772dc30eab26742eb2780f2c2bfa"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.605511 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "36201a0278ff4d648f657182295459c6"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "3d30b08170923dd437fbf6e0ce2bb219"
server_key_iv: "7aff772dc30eab26742eb2780f2c2bfa"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.609620 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:21.612378 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:21.613773 22271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:21.613900 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "36201a0278ff4d648f657182295459c6"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "3d30b08170923dd437fbf6e0ce2bb219"
server_key_iv: "7aff772dc30eab26742eb2780f2c2bfa"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.614018 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:21.629685 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:21.633002 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:21.633214 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:21.641492 22271 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40407
I20260810 14:35:21.641518 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40407 every 8 connection(s)
I20260810 14:35:21.642601 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:21.645653 22344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:21.649744 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22271
I20260810 14:35:21.649852 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:21.650138 21677 external_mini_cluster.cc:1428] Setting key 171a9aab5ab817fe1dd1dccae4019833
I20260810 14:35:21.652395 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: Bootstrap starting.
I20260810 14:35:21.655412 22344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:21.656349 22344 log.cc:824] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:21.659008 22344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: No bootstrap required, opened a new log
Aug 10 14:35:21 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:21.663618 22344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } }
I20260810 14:35:21.663950 22344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:21.664018 22344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36201a0278ff4d648f657182295459c6, State: Initialized, Role: FOLLOWER
I20260810 14:35:21.664593 22344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } }
I20260810 14:35:21.664737 22344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:21.664829 22344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:21.664995 22344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:21.666142 22344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } }
I20260810 14:35:21.666534 22344 leader_election.cc:302] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36201a0278ff4d648f657182295459c6; no voters:
I20260810 14:35:21.667001 22344 leader_election.cc:288] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:21.667214 22349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:21.667552 22349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [term 1 LEADER]: Becoming Leader. State: Replica: 36201a0278ff4d648f657182295459c6, State: Running, Role: LEADER
I20260810 14:35:21.667953 22349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } }
I20260810 14:35:21.668701 22344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:21.668761 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:21.651714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.21.43.126:40407)
0810 14:35:21.652216 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:21.652236 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:21.652283 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:35:21.653163 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:21.653187 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:21.653641 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:21.654050 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:21.654812 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.656353 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.657563 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.658248 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.658425 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:21.661397 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:21.661404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:21.661423 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:21.661461 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:21.664340 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:21.664931 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:21.664937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:21.664942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:21.665074 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:21.665559 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:21.665565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:21.665566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:21.666416 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:21.666741 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:21.667217 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 14:35:21.667635 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":145,"threads_started":1}
I20260810 14:35:21.669951 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36201a0278ff4d648f657182295459c6. Latest consensus state: current_term: 1 leader_uuid: "36201a0278ff4d648f657182295459c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } } }
I20260810 14:35:21.670080 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:21.670441 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36201a0278ff4d648f657182295459c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36201a0278ff4d648f657182295459c6" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40407 } } }
I20260810 14:35:21.670539 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6 [sys.catalog]: This master's current role is: LEADER
W20260810 14:35:21.673980 22367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:35:21.674057 22367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:35:21.674142 22361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:21.675179 22361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:21.681596 22361 catalog_manager.cc:1347] Generated new cluster ID: b2eb5fd786284e839e2a8a32392c2067
I20260810 14:35:21.681686 22361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:21.704566 22361 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:21.705960 22361 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:21.715696 22361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36201a0278ff4d648f657182295459c6: Generated new TSK 0
I20260810 14:35:21.716619 22361 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:21.757048 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:21.884804 22375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:21.885149 22375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:21.885231 22375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:21.889532 22375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:21.889658 22375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:21.889770 22375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:21.895840 22375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:21.897099 22375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:21.898324 22375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:21.909137 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:21.909303 22375 server_base.cc:1034] running on GCE node
W20260810 14:35:21.909168 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:21.909440 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:21.909946 22375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:21.910681 22375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:21.911886 22375 hybrid_clock.cc:648] HybridClock initialized: now 1786372521911869 us; error 34 us; skew 500 ppm
Aug 10 14:35:21 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:21.915007 22375 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:21.916163 22375 webserver.cc:460] Webserver started at http://127.21.43.65:35603/ using document root <none> and password file <none>
I20260810 14:35:21.916790 22375 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:21.916889 22375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:21.917187 22375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:21.919780 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "8f197837f48d42c9a6a0685517a956a3"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "9e1a56e5e312fda9bf4e2bd0348c7282"
server_key_iv: "8af993eb0fc163e938d7390480dd5dd0"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.920410 22375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "8f197837f48d42c9a6a0685517a956a3"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "9e1a56e5e312fda9bf4e2bd0348c7282"
server_key_iv: "8af993eb0fc163e938d7390480dd5dd0"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.924705 22375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:21.927587 22391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:21.929205 22375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:35:21.929395 22375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "8f197837f48d42c9a6a0685517a956a3"
format_stamp: "Formatted at 2026-08-10 14:35:21 on dist-test-slave-chsk"
server_key: "9e1a56e5e312fda9bf4e2bd0348c7282"
server_key_iv: "8af993eb0fc163e938d7390480dd5dd0"
server_key_version: "encryptionkey@0"
I20260810 14:35:21.929548 22375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:21.947803 22375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:21.951522 22375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:21.951788 22375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:21.952718 22375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:21.954039 22375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:21.954126 22375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:21.954187 22375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:21.954214 22375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:21.964322 22375 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:38335
I20260810 14:35:21.964341 22514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:38335 every 8 connection(s)
I20260810 14:35:21.965327 22375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:21.966225 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22375
I20260810 14:35:21.966321 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:21.966611 21677 external_mini_cluster.cc:1428] Setting key b4307ccfc938d783956401fa1ea658a8
Aug 10 14:35:21 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:21.982784 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:21.967373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:33989 (local address 127.21.43.126:40407)
0810 14:35:21.967538 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:21.967543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:21.968598 (+ 1055us) server_negotiation.cc:407] Connection header received
0810 14:35:21.970114 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:21.970121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:21.970193 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:21.970297 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:21.972263 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.972802 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.974862 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.975372 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.975494 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:21.978718 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:21.978721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:21.978723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:21.978749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:21.980341 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:21.981024 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:21.981031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:21.981033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:21.981154 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:21.981568 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:21.981572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:21.981574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:21.981784 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:21.981899 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:21.982450 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 14:35:21.982603 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:21.983948 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:21.967876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40407 (local address 127.21.43.65:33989)
0810 14:35:21.968400 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:21.968505 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:21.969875 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:21.970534 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:21.970545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:21.971160 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:21.972046 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:21.972072 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.972946 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.972953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:21.974589 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:21.974607 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:21.975587 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:21.975595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:21.975852 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:21.976611 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:21.976634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:21.978497 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:21.980492 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:21.980498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:21.980575 (+ 77us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:21.980909 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:21.981252 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:21.981255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:21.981257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:21.981461 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:21.981898 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:21.981907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:21.982204 (+ 297us) client_negotiation.cc:769] Sending connection context
0810 14:35:21.982408 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 14:35:21.982766 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":161,"threads_started":1}
I20260810 14:35:21.985451 22515 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40407
I20260810 14:35:21.985736 22515 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:21.986421 22515 heartbeater.cc:511] Master 127.21.43.126:40407 requested a full tablet report, sending...
I20260810 14:35:21.988312 22304 ts_manager.cc:194] Registered new tserver with Master: 8f197837f48d42c9a6a0685517a956a3 (127.21.43.65:38335)
I20260810 14:35:21.989537 22304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:33989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:22.002416 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:22.125671 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:22.126166 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:22.126258 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:22.131850 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:22.131958 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:22.132071 22520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:22.139353 22520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:22.140538 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:22.141554 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:22.152227 22528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:22.152226 22527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:22.152604 22520 server_base.cc:1034] running on GCE node
W20260810 14:35:22.152230 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:22.153092 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:22.153843 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:22.155179 22520 hybrid_clock.cc:648] HybridClock initialized: now 1786372522155107 us; error 59 us; skew 500 ppm
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:22.158435 22520 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:22.159502 22520 webserver.cc:460] Webserver started at http://127.21.43.66:37281/ using document root <none> and password file <none>
I20260810 14:35:22.160111 22520 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:22.160177 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:22.160356 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:22.162586 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "3bc4935300b74534bce91af8832975d9"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "5fda2c334d48e5827b624c265f713b6b"
server_key_iv: "5c241a6f0a534b17de2e6aa6fdd2b0a1"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.163215 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "3bc4935300b74534bce91af8832975d9"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "5fda2c334d48e5827b624c265f713b6b"
server_key_iv: "5c241a6f0a534b17de2e6aa6fdd2b0a1"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.167676 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:22.171628 22538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.173164 22520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:35:22.173303 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "3bc4935300b74534bce91af8832975d9"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "5fda2c334d48e5827b624c265f713b6b"
server_key_iv: "5c241a6f0a534b17de2e6aa6fdd2b0a1"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.173426 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:22.190037 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:22.193240 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:22.193406 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:22.193960 22520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:22.194775 22520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:22.194824 22520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.194866 22520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:22.194888 22520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.204259 22520 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:46107
I20260810 14:35:22.204277 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:46107 every 8 connection(s)
I20260810 14:35:22.205212 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:22.210955 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22520
I20260810 14:35:22.211072 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:22.211380 21677 external_mini_cluster.cc:1428] Setting key 75f006196762cfa85148660c755b1141
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:22.221426 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.207065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48977 (local address 127.21.43.126:40407)
0810 14:35:22.207239 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.207243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.208165 (+ 922us) server_negotiation.cc:407] Connection header received
0810 14:35:22.209436 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.209439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.209487 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.209565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.211047 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.211872 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.213173 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.213393 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.213477 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.216963 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.216970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.216976 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.217012 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.218974 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.219570 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.219578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.219581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.219683 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.220086 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.220092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.220096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.220392 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.220508 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.221098 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.221239 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:35:22.222326 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.207503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40407 (local address 127.21.43.66:48977)
0810 14:35:22.207987 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.208051 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.209225 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.209730 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.209738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.210128 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:22.210855 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.210880 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.212015 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.212018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.213051 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.213061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.213556 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.213562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.213803 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.214511 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:22.214540 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:22.216761 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:22.219107 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:22.219112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:22.219126 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:22.219457 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:22.219818 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:22.219820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:22.219821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:22.219972 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:22.220527 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:22.220533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:22.220805 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.221011 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.221380 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:35:22.223664 22660 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40407
I20260810 14:35:22.223912 22660 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:22.224658 22660 heartbeater.cc:511] Master 127.21.43.126:40407 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:22.225867 22304 ts_manager.cc:194] Registered new tserver with Master: 3bc4935300b74534bce91af8832975d9 (127.21.43.66:46107)
I20260810 14:35:22.226701 22304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:48977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:22.242211 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:22.361109 22666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:22.361428 22666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:22.361505 22666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:22.365622 22666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:22.365738 22666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:22.365836 22666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:22.371747 22666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40407
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:22.372946 22666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:22.374043 22666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:22.384565 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:22.384629 22666 server_base.cc:1034] running on GCE node
W20260810 14:35:22.384572 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:22.384565 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:22.385273 22666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:22.386016 22666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:22.387213 22666 hybrid_clock.cc:648] HybridClock initialized: now 1786372522387184 us; error 50 us; skew 500 ppm
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:22.390010 22666 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:22.390945 22666 webserver.cc:460] Webserver started at http://127.21.43.67:32837/ using document root <none> and password file <none>
I20260810 14:35:22.391600 22666 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:22.391698 22666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:22.391942 22666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:22.394063 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "7243247240a84dada716c095a8b1b5c3"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "7d4fa64ebd86721e1f05a6b85426570e"
server_key_iv: "adfacd3a288177022716e1b9965081f3"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.394652 22666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "7243247240a84dada716c095a8b1b5c3"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "7d4fa64ebd86721e1f05a6b85426570e"
server_key_iv: "adfacd3a288177022716e1b9965081f3"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.398610 22666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:22.401485 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.402841 22666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:22.403014 22666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "7243247240a84dada716c095a8b1b5c3"
format_stamp: "Formatted at 2026-08-10 14:35:22 on dist-test-slave-chsk"
server_key: "7d4fa64ebd86721e1f05a6b85426570e"
server_key_iv: "adfacd3a288177022716e1b9965081f3"
server_key_version: "encryptionkey@0"
I20260810 14:35:22.403180 22666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:22.421167 22666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:22.424474 22666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:22.424706 22666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:22.425352 22666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:22.426232 22666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:22.426307 22666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.426381 22666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:22.426434 22666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:22.437489 22666 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:37765
I20260810 14:35:22.437508 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:37765 every 8 connection(s)
I20260810 14:35:22.438484 22666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:22.440128 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22666
I20260810 14:35:22.440235 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:22.440575 21677 external_mini_cluster.cc:1428] Setting key 57658c6497ac5834352f8c927e0c7d24
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:22.452818 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.440649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:35827 (local address 127.21.43.126:40407)
0810 14:35:22.440813 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.440818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.441977 (+ 1159us) server_negotiation.cc:407] Connection header received
0810 14:35:22.443639 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.443642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.443689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.443758 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.444944 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.445474 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.446595 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.446806 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.446883 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.449419 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.449422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.449424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.449448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.450902 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.451442 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.451444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.451446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.451520 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.451829 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.451831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.451832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.452010 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.452083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.452566 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.452686 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:35:22.453780 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.441137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40407 (local address 127.21.43.67:35827)
0810 14:35:22.441805 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.441865 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.443428 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.443923 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.443930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.444312 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:22.444816 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.444829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.445590 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.445592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.446475 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.446485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.446944 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.446947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.447089 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.447744 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:22.447761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:22.449247 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:22.451007 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:22.451012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:22.451025 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:22.451353 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:22.451603 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:22.451605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:22.451606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:22.451740 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:22.452100 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:22.452105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:22.452346 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.452501 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.452786 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":240,"threads_started":1}
I20260810 14:35:22.455212 22808 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40407
I20260810 14:35:22.455488 22808 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:22.456013 22808 heartbeater.cc:511] Master 127.21.43.126:40407 requested a full tablet report, sending...
I20260810 14:35:22.457139 22303 ts_manager.cc:194] Registered new tserver with Master: 7243247240a84dada716c095a8b1b5c3 (127.21.43.67:37765)
I20260810 14:35:22.457715 22303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:35827
I20260810 14:35:22.467310 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:22.479434 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.471687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.21.43.126:40407)
0810 14:35:22.471821 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.471830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.472000 (+ 170us) server_negotiation.cc:407] Connection header received
0810 14:35:22.472119 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.472121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.472165 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.472240 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.472938 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.473447 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.474302 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.474500 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.474575 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.476186 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.476189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.476191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.476216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.477768 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.478282 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.478284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.478286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.478363 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.478594 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.478596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.478597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.478804 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.478906 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.479099 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.479251 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:35:22.484575 22303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:22.487970 22303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:22.503641 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.497356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.21.43.65:38335)
0810 14:35:22.498439 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.498445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.498469 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:35:22.498553 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.498559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.498775 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.498933 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:22.499349 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500538 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.502273 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.502832 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.502961 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503125 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.503410 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.503517 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":161,"threads_started":1}
I20260810 14:35:22.503626 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.497233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38335 (local address 127.0.0.1:52680)
0810 14:35:22.497917 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.497968 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.498107 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.498948 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.498953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.498980 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:22.499186 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.499195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500676 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.500681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.502101 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.502112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.502975 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.502979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.503096 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503225 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.503335 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.503461 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":26,"thread_start_us":208,"threads_started":1}
I20260810 14:35:22.503830 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.497682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.21.43.66:46107)
0810 14:35:22.498398 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.498410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.498946 (+ 536us) server_negotiation.cc:407] Connection header received
0810 14:35:22.499281 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.499286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.499448 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.499549 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:22.499883 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500660 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.502240 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.502997 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.503108 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503283 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.503573 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.503664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":134,"threads_started":1}
I20260810 14:35:22.503981 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.498221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.21.43.67:37765)
0810 14:35:22.498666 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.498676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.498698 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:35:22.498812 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.498819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.498984 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.499477 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:22.499641 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500701 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.502091 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.502938 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.503193 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503315 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.503721 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.503812 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":142,"threads_started":1}
I20260810 14:35:22.504133 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.498065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:37765 (local address 127.0.0.1:47168)
0810 14:35:22.498612 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.498625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.498707 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.499199 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.499204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.499226 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:22.499318 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.499325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500871 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.500878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.501938 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.501948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.503094 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.503097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.503460 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503497 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.503573 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.503646 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":217,"thread_start_us":104,"threads_started":1}
I20260810 14:35:22.504437 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.497279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46107 (local address 127.0.0.1:37502)
0810 14:35:22.498842 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.498852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.499077 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.499698 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.499700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.499715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:22.499772 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.499775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.500972 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.500976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.502118 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.502124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.503233 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.503237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.503335 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.503429 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.504054 (+ 625us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.504138 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":51,"thread_start_us":124,"threads_started":1}
I20260810 14:35:22.507267 22589 tablet_service.cc:1467] Processing CreateTablet for tablet de90e89921a344869b67eb4aa9fa825c (DEFAULT_TABLE table=test-table [id=b36d3907f83e4ff998335e62df089a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:22.507268 22738 tablet_service.cc:1467] Processing CreateTablet for tablet de90e89921a344869b67eb4aa9fa825c (DEFAULT_TABLE table=test-table [id=b36d3907f83e4ff998335e62df089a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:22.507228 22443 tablet_service.cc:1467] Processing CreateTablet for tablet de90e89921a344869b67eb4aa9fa825c (DEFAULT_TABLE table=test-table [id=b36d3907f83e4ff998335e62df089a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:22.508452 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90e89921a344869b67eb4aa9fa825c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:22.508452 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90e89921a344869b67eb4aa9fa825c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:22.508455 22443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90e89921a344869b67eb4aa9fa825c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:22.515676 22828 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Bootstrap starting.
I20260810 14:35:22.516472 22829 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Bootstrap starting.
I20260810 14:35:22.516568 22830 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Bootstrap starting.
I20260810 14:35:22.518447 22828 tablet_bootstrap.cc:654] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:22.518471 22829 tablet_bootstrap.cc:654] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:22.519177 22830 tablet_bootstrap.cc:654] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:22.519501 22828 log.cc:824] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:22.519724 22829 log.cc:824] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:22.520059 22830 log.cc:824] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:22.521831 22828 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: No bootstrap required, opened a new log
I20260810 14:35:22.521934 22829 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: No bootstrap required, opened a new log
I20260810 14:35:22.522102 22828 ts_tablet_manager.cc:1397] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:35:22.522137 22830 tablet_bootstrap.cc:492] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: No bootstrap required, opened a new log
I20260810 14:35:22.522153 22829 ts_tablet_manager.cc:1397] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:22.522296 22830 ts_tablet_manager.cc:1397] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:22.525681 22828 raft_consensus.cc:348] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.525952 22828 raft_consensus.cc:374] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:22.526015 22828 raft_consensus.cc:729] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f197837f48d42c9a6a0685517a956a3, State: Initialized, Role: FOLLOWER
I20260810 14:35:22.526105 22830 raft_consensus.cc:348] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.526105 22829 raft_consensus.cc:348] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.526373 22829 raft_consensus.cc:374] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:22.526373 22830 raft_consensus.cc:374] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:22.526434 22829 raft_consensus.cc:729] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7243247240a84dada716c095a8b1b5c3, State: Initialized, Role: FOLLOWER
I20260810 14:35:22.526434 22830 raft_consensus.cc:729] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc4935300b74534bce91af8832975d9, State: Initialized, Role: FOLLOWER
I20260810 14:35:22.526571 22828 consensus_queue.cc:260] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.526870 22830 consensus_queue.cc:260] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.526870 22829 consensus_queue.cc:260] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.527416 22515 heartbeater.cc:503] Master 127.21.43.126:40407 was elected leader, sending a full tablet report...
I20260810 14:35:22.527791 22828 ts_tablet_manager.cc:1428] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:35:22.527906 22830 ts_tablet_manager.cc:1428] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 14:35:22.527906 22829 ts_tablet_manager.cc:1428] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260810 14:35:22.528250 22660 heartbeater.cc:503] Master 127.21.43.126:40407 was elected leader, sending a full tablet report...
I20260810 14:35:22.528316 22808 heartbeater.cc:503] Master 127.21.43.126:40407 was elected leader, sending a full tablet report...
I20260810 14:35:22.630564 22836 raft_consensus.cc:482] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:22.630826 22836 raft_consensus.cc:504] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.632447 22836 leader_election.cc:288] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f197837f48d42c9a6a0685517a956a3 (127.21.43.65:38335), 3bc4935300b74534bce91af8832975d9 (127.21.43.66:46107)
I20260810 14:35:22.636246 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.632894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:51021 (local address 127.21.43.66:46107)
0810 14:35:22.633021 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.633025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.633040 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:35:22.633160 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.633166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.633224 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.633313 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:22.633592 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.634135 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.635244 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.635666 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.635803 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.635863 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.636050 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.636121 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:22.636246 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.632699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46107 (local address 127.21.43.67:51021)
0810 14:35:22.632926 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.632942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.633053 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.633325 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.633328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.633346 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:22.633471 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.633477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.634258 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.634262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.635071 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.635080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.635799 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.635802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.635910 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.635932 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.636031 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.636109 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260810 14:35:22.636713 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.632730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38335 (local address 127.21.43.67:37205)
0810 14:35:22.633187 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:22.633200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:22.633341 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:22.633589 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:22.633592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:22.633608 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:22.633743 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.633750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.634595 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.634598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.635514 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:22.635523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.636209 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.636212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:22.636314 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.636335 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:35:22.636460 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:35:22.636526 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":143,"threads_started":1}
I20260810 14:35:22.636714 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.632851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:37205 (local address 127.21.43.65:38335)
0810 14:35:22.632989 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.632994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.633296 (+ 302us) server_negotiation.cc:407] Connection header received
0810 14:35:22.633430 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.633433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.633485 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.633576 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:22.633863 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.634479 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.635649 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.636079 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.636215 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.636264 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.636502 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.636589 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:22.637073 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90e89921a344869b67eb4aa9fa825c" candidate_uuid: "7243247240a84dada716c095a8b1b5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc4935300b74534bce91af8832975d9" is_pre_election: true
I20260810 14:35:22.637346 22464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90e89921a344869b67eb4aa9fa825c" candidate_uuid: "7243247240a84dada716c095a8b1b5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f197837f48d42c9a6a0685517a956a3" is_pre_election: true
I20260810 14:35:22.637543 22612 raft_consensus.cc:2454] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7243247240a84dada716c095a8b1b5c3 in term 0.
I20260810 14:35:22.637656 22464 raft_consensus.cc:2454] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7243247240a84dada716c095a8b1b5c3 in term 0.
I20260810 14:35:22.638209 22686 leader_election.cc:302] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc4935300b74534bce91af8832975d9, 7243247240a84dada716c095a8b1b5c3; no voters:
I20260810 14:35:22.638518 22836 raft_consensus.cc:2781] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:22.638615 22836 raft_consensus.cc:482] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:22.638669 22836 raft_consensus.cc:3037] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:22.639801 22836 raft_consensus.cc:504] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.640236 22836 leader_election.cc:288] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8f197837f48d42c9a6a0685517a956a3 (127.21.43.65:38335), 3bc4935300b74534bce91af8832975d9 (127.21.43.66:46107)
I20260810 14:35:22.640632 22464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90e89921a344869b67eb4aa9fa825c" candidate_uuid: "7243247240a84dada716c095a8b1b5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f197837f48d42c9a6a0685517a956a3"
I20260810 14:35:22.640753 22612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90e89921a344869b67eb4aa9fa825c" candidate_uuid: "7243247240a84dada716c095a8b1b5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc4935300b74534bce91af8832975d9"
I20260810 14:35:22.640781 22464 raft_consensus.cc:3037] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:22.640923 22612 raft_consensus.cc:3037] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:22.642036 22464 raft_consensus.cc:2454] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7243247240a84dada716c095a8b1b5c3 in term 1.
I20260810 14:35:22.642287 22612 raft_consensus.cc:2454] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7243247240a84dada716c095a8b1b5c3 in term 1.
I20260810 14:35:22.642383 22687 leader_election.cc:302] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7243247240a84dada716c095a8b1b5c3, 8f197837f48d42c9a6a0685517a956a3; no voters:
I20260810 14:35:22.642724 22836 raft_consensus.cc:2781] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:22.643110 22836 raft_consensus.cc:686] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [term 1 LEADER]: Becoming Leader. State: Replica: 7243247240a84dada716c095a8b1b5c3, State: Running, Role: LEADER
I20260810 14:35:22.643793 22836 consensus_queue.cc:237] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } }
I20260810 14:35:22.647661 22303 catalog_manager.cc:5495] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7243247240a84dada716c095a8b1b5c3 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "7243247240a84dada716c095a8b1b5c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f197837f48d42c9a6a0685517a956a3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc4935300b74534bce91af8832975d9" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7243247240a84dada716c095a8b1b5c3" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37765 } health_report { overall_health: HEALTHY } } }
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:22.673298 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.663501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.21.43.65:38335)
0810 14:35:22.663665 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.663668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.663796 (+ 128us) server_negotiation.cc:407] Connection header received
0810 14:35:22.663903 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.663911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.663959 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.664032 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.664698 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.665228 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.666099 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.666310 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.666393 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.669164 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.669178 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.669195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.669251 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.671426 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.671901 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.671912 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.671913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.672003 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.672300 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.672304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.672306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.672583 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.672818 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.672991 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.673124 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260810 14:35:22.689723 22810 tablet.cc:2369] T de90e89921a344869b67eb4aa9fa825c P 7243247240a84dada716c095a8b1b5c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:22.704185 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.694937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.21.43.65:38335)
0810 14:35:22.695211 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.695216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.695334 (+ 118us) server_negotiation.cc:407] Connection header received
0810 14:35:22.695451 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.695453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.695507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.695590 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.696345 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.696895 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.697760 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.697988 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.698075 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.700758 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.700765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.700769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.700806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.702365 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.702811 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.702814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.702815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.702892 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.703283 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.703285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.703286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.703457 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.703562 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.703923 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.704042 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260810 14:35:22.706395 22661 tablet.cc:2369] T de90e89921a344869b67eb4aa9fa825c P 3bc4935300b74534bce91af8832975d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:22.716840 22516 tablet.cc:2369] T de90e89921a344869b67eb4aa9fa825c P 8f197837f48d42c9a6a0685517a956a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:22.736734 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:22.727569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.21.43.65:38335)
0810 14:35:22.727739 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:22.727744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:22.727938 (+ 194us) server_negotiation.cc:407] Connection header received
0810 14:35:22.728131 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:22.728137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:22.728204 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:22.728299 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:22.729145 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.729656 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.730531 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:22.730754 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:22.730868 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:22.733508 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:22.733512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:22.733514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:22.733543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:22.734891 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.735438 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.735441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.735443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.735521 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:22.735813 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:22.735817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:22.735819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:22.736006 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:22.736162 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:22.736419 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 14:35:22.736549 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 14:35:22.738341 22405 tablet_service.cc:2908] Rejecting scan request for tablet de90e89921a344869b67eb4aa9fa825c: Uninitialized: clean time has not yet been initialized
I20260810 14:35:22.740530 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22375
I20260810 14:35:22.750129 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22520
I20260810 14:35:22.758919 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22666
I20260810 14:35:22.767336 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22271
2026-08-10T14:35:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2572 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22853](info): set up 2 sockets
Aug 10 14:35:22 dist-test-slave-chsk krb5kdc[22853](info): commencing operation
krb5kdc: starting...
W20260810 14:35:23.878873 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:23 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:24.024550 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38833
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:24.142802 22869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:24.143107 22869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:24.143231 22869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:24.147454 22869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:24.147531 22869 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:35:24.147555 22869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:24.147575 22869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:24.147591 22869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:24.147655 22869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:24.153340 22869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38833
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:24.154516 22869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:24.155650 22869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:24.164471 22874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.164515 22875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.164670 22877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:24.164965 22869 server_base.cc:1034] running on GCE node
I20260810 14:35:24.165503 22869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:24.166610 22869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:24.167788 22869 hybrid_clock.cc:648] HybridClock initialized: now 1786372524167770 us; error 26 us; skew 500 ppm
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:24.170631 22869 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:24.171703 22869 webserver.cc:460] Webserver started at http://127.21.43.126:33857/ using document root <none> and password file <none>
I20260810 14:35:24.172264 22869 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:24.172327 22869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:24.172503 22869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:24.174614 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "7be47b7994754ae7bf3435ec520f0ae0"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "56f1194aa8d434caaaa2d75443add9f8"
server_key_iv: "328ace6b6853fab0600aecc2559406e3"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.175248 22869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "7be47b7994754ae7bf3435ec520f0ae0"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "56f1194aa8d434caaaa2d75443add9f8"
server_key_iv: "328ace6b6853fab0600aecc2559406e3"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.179247 22869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:24.181840 22885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.183303 22869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:35:24.183429 22869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "7be47b7994754ae7bf3435ec520f0ae0"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "56f1194aa8d434caaaa2d75443add9f8"
server_key_iv: "328ace6b6853fab0600aecc2559406e3"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.183532 22869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:24.197942 22869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:24.201550 22869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:24.201730 22869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:24.211437 22869 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38833
I20260810 14:35:24.211500 22949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38833 every 8 connection(s)
I20260810 14:35:24.212566 22869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:24.215513 22950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:24.221366 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Bootstrap starting.
I20260810 14:35:24.222585 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22869
I20260810 14:35:24.222695 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:24.222987 21677 external_mini_cluster.cc:1428] Setting key 7cdb336082fe1ee08088fd7e6987f3d2
I20260810 14:35:24.224448 22950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:24.225445 22950 log.cc:824] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:24.227905 22950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: No bootstrap required, opened a new log
I20260810 14:35:24.232116 22950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } }
I20260810 14:35:24.232375 22950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:24.232451 22950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be47b7994754ae7bf3435ec520f0ae0, State: Initialized, Role: FOLLOWER
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:24.232954 22950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } }
I20260810 14:35:24.233096 22950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:24.233187 22950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:24.233325 22950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:24.234486 22950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } }
I20260810 14:35:24.234802 22950 leader_election.cc:302] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be47b7994754ae7bf3435ec520f0ae0; no voters:
I20260810 14:35:24.235071 22950 leader_election.cc:288] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:24.235234 22956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:24.235497 22956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 7be47b7994754ae7bf3435ec520f0ae0, State: Running, Role: LEADER
I20260810 14:35:24.235812 22956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } }
I20260810 14:35:24.236435 22950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:24.237804 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7be47b7994754ae7bf3435ec520f0ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } } }
I20260810 14:35:24.237941 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:24.238278 22958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be47b7994754ae7bf3435ec520f0ae0. Latest consensus state: current_term: 1 leader_uuid: "7be47b7994754ae7bf3435ec520f0ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be47b7994754ae7bf3435ec520f0ae0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38833 } } }
I20260810 14:35:24.238368 22958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:24.239418 22965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:24.242632 22965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:24.245239 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.224291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.21.43.126:38833)
0810 14:35:24.224842 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.224860 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.224905 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:35:24.225778 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.225808 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.226185 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.226568 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.227358 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.228815 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.229707 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.230478 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.230657 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.233494 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:24.233500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:24.233518 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:24.233560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:24.241630 (+ 8070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.242329 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.242336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.242342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.242493 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.242831 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.242834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.242836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.243412 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:24.243639 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:24.243926 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 14:35:24.244240 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":160,"threads_started":1}
I20260810 14:35:24.249374 22965 catalog_manager.cc:1347] Generated new cluster ID: f6da4d7efd174cf2b7b0eb7c71839588
I20260810 14:35:24.249462 22965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:24.272787 22965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:24.273792 22965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:24.284852 22965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Generated new TSK 0
I20260810 14:35:24.285719 22965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:24.323096 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38833
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:24.446135 22979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:24.446431 22979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:24.446480 22979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:24.450585 22979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:24.450650 22979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:24.450691 22979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:24.450726 22979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:24.456153 22979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:24.457235 22979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:24.458258 22979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:24.468532 22987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.468526 22984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.468588 22985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:24.468835 22979 server_base.cc:1034] running on GCE node
I20260810 14:35:24.469357 22979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:24.470139 22979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:24.471342 22979 hybrid_clock.cc:648] HybridClock initialized: now 1786372524471312 us; error 43 us; skew 500 ppm
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:24.474488 22979 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:24.475628 22979 webserver.cc:460] Webserver started at http://127.21.43.65:45951/ using document root <none> and password file <none>
I20260810 14:35:24.476264 22979 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:24.476365 22979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:24.476621 22979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:24.478883 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "df0dc7e08fb8481bb76192066c68ecae"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "3a1002d9dbd792ced792d056128c38c8"
server_key_iv: "8f9a64077519136ed815be8219bc21c7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.479668 22979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "df0dc7e08fb8481bb76192066c68ecae"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "3a1002d9dbd792ced792d056128c38c8"
server_key_iv: "8f9a64077519136ed815be8219bc21c7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.483915 22979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:24.486639 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.488140 22979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:35:24.488272 22979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "df0dc7e08fb8481bb76192066c68ecae"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "3a1002d9dbd792ced792d056128c38c8"
server_key_iv: "8f9a64077519136ed815be8219bc21c7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.488395 22979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:24.523026 22979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:24.526211 22979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:24.526386 22979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:24.527977 22979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:24.528067 22979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.528167 22979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:24.528204 22979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:24.539615 22979 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39407
I20260810 14:35:24.539695 23117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39407 every 8 connection(s)
I20260810 14:35:24.540876 22979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:24.542138 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 22979
I20260810 14:35:24.542269 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:24.542572 21677 external_mini_cluster.cc:1428] Setting key 103a28f3f1fdb8e4fdb8fa7c38a612e2
I20260810 14:35:24.544034 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.528790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:41865 (local address 127.21.43.126:38833)
0810 14:35:24.528954 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.528958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.529904 (+ 946us) server_negotiation.cc:407] Connection header received
0810 14:35:24.530973 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.530978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.531050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.531170 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.533118 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.533780 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.535864 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.536293 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.536428 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.539811 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:24.539815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:24.539818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:24.539849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:24.541483 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.542219 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.542225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.542228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.542338 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.542735 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.542738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.542739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.542933 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:24.543052 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:24.543715 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 14:35:24.543860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:35:24.544879 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.529246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.65:41865)
0810 14:35:24.529722 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:24.529774 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:24.530712 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:24.531657 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:24.531664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:24.532188 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:24.532924 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.532950 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.533923 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.533928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.535699 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.535714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.536475 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.536480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.536728 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.537418 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:24.537445 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:24.539547 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:24.541637 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.541644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.541660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.542071 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.542431 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.542435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.542437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.542619 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.543061 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:24.543069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:24.543426 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 14:35:24.543626 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 14:35:24.543974 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260810 14:35:24.546291 23119 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38833
I20260810 14:35:24.546478 23119 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:24.547101 23119 heartbeater.cc:511] Master 127.21.43.126:38833 requested a full tablet report, sending...
I20260810 14:35:24.549856 22902 ts_manager.cc:194] Registered new tserver with Master: df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407)
I20260810 14:35:24.551712 22902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:41865
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:24.556443 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.548826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:55083 (local address 127.21.43.126:38833)
0810 14:35:24.548934 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.548938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.549302 (+ 364us) server_negotiation.cc:407] Connection header received
0810 14:35:24.549448 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.549451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.549497 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.549589 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.550066 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.550583 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.551479 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.551731 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.551843 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.553131 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:24.553135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:24.553140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:24.553172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:24.554942 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.555451 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.555454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.555456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.555552 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.555787 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.555789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.555790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.555945 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:24.556016 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:24.556221 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 14:35:24.556318 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":33}
I20260810 14:35:24.556622 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.548766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.65:55083)
0810 14:35:24.549209 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:24.549222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:24.549343 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:24.549604 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:24.549607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:24.549819 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:24.549914 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.549919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.550718 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.550721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.551356 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.551365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.551849 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.551853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.551953 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.552526 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:24.552539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:24.552969 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:24.555042 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.555045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.555046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.555312 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.555638 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.555640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.555642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.555723 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.556021 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:24.556023 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:24.556096 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 14:35:24.556186 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:35:24.556336 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":252,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:24.574427 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38833
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:24.690038 23128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:24.690325 23128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:24.690404 23128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:24.694759 23128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:24.694831 23128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:24.694893 23128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:24.694968 23128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:24.700407 23128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:24.701545 23128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:24.702620 23128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:24.712945 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.712957 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.713028 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:24.713228 23128 server_base.cc:1034] running on GCE node
I20260810 14:35:24.713667 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:24.714391 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:24.715590 23128 hybrid_clock.cc:648] HybridClock initialized: now 1786372524715555 us; error 46 us; skew 500 ppm
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:24.718463 23128 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:24.719472 23128 webserver.cc:460] Webserver started at http://127.21.43.66:40057/ using document root <none> and password file <none>
I20260810 14:35:24.720057 23128 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:24.720167 23128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:24.720403 23128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:24.722615 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "d33201f03d3d4a4187dea597b5561d25"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "089e94294cc1ad6dbfdda2715dbdac5d"
server_key_iv: "a32acdc937a0652647e710ba5b44d35d"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.723304 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "d33201f03d3d4a4187dea597b5561d25"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "089e94294cc1ad6dbfdda2715dbdac5d"
server_key_iv: "a32acdc937a0652647e710ba5b44d35d"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.727625 23128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:35:24.730669 23144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.732204 23128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:24.732358 23128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "d33201f03d3d4a4187dea597b5561d25"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "089e94294cc1ad6dbfdda2715dbdac5d"
server_key_iv: "a32acdc937a0652647e710ba5b44d35d"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.732488 23128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:24.744351 23128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:24.747891 23128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:24.748119 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:24.749764 23128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:24.749848 23128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.749940 23128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:24.749992 23128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:24.761340 23128 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:35713
I20260810 14:35:24.761410 23271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:35713 every 8 connection(s)
I20260810 14:35:24.762363 23128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:24.764108 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.750508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:52277 (local address 127.21.43.126:38833)
0810 14:35:24.750697 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.750704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.752268 (+ 1564us) server_negotiation.cc:407] Connection header received
0810 14:35:24.753763 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.753766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.753835 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.753955 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.755169 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.755713 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.756851 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.757087 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.757229 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.760016 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:24.760020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:24.760023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:24.760058 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:24.761797 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.762332 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.762336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.762338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.762426 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.762908 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.762911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.762913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.763227 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:24.763342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:24.763876 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 14:35:24.763984 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:35:24.764886 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.751452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.66:52277)
0810 14:35:24.752056 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:24.752119 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:24.753487 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:24.754148 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:24.754155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:24.754477 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:24.754984 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.755004 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.755840 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.755844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.756720 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.756730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.757285 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.757290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.757457 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.758107 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:24.758130 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:24.759845 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:24.761917 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.761923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.761937 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.762220 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.762566 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.762569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.762571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.762767 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.763398 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:24.763404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:24.763647 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 14:35:24.763809 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 14:35:24.764062 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260810 14:35:24.765998 23273 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38833
I20260810 14:35:24.766239 23273 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:24.766729 23273 heartbeater.cc:511] Master 127.21.43.126:38833 requested a full tablet report, sending...
I20260810 14:35:24.767845 22903 ts_manager.cc:194] Registered new tserver with Master: d33201f03d3d4a4187dea597b5561d25 (127.21.43.66:35713)
I20260810 14:35:24.768440 22903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:52277
I20260810 14:35:24.772403 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23128
I20260810 14:35:24.772547 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:24.772891 21677 external_mini_cluster.cc:1428] Setting key 22b4be0366eb874795f7885b77978677
I20260810 14:35:24.775745 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.768517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:51623 (local address 127.21.43.126:38833)
0810 14:35:24.768632 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.768636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.768848 (+ 212us) server_negotiation.cc:407] Connection header received
0810 14:35:24.768926 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.768929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.768976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.769059 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.769579 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.770048 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.770964 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.771250 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.771334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.772446 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:24.772453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:24.772459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:24.772487 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:24.773950 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.774449 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.774452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.774454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.774546 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:24.774851 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:24.774853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:24.774854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:24.775056 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:24.775209 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:24.775520 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 14:35:24.775627 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:24.775949 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.768439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.66:51623)
0810 14:35:24.768768 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:24.768780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:24.768863 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:24.769119 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:24.769122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:24.769344 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:24.769457 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.769464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.770206 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.770210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.770862 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.770871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.771355 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.771358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.771426 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.771870 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:24.771882 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:24.772278 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:24.774083 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.774086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.774088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.774346 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.774651 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:24.774654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:24.774656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:24.774755 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:24.775206 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:24.775212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:24.775332 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 14:35:24.775459 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:35:24.775647 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:24.803025 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38833
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:24.920581 23282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:24.920885 23282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:24.920933 23282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:24.925228 23282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:24.925324 23282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:24.925369 23282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:24.925403 23282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:24.931209 23282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:24.932372 23282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:24.933413 23282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:24.943830 23291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:24.943970 23282 server_base.cc:1034] running on GCE node
W20260810 14:35:24.943826 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:24.944211 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:24.944633 23282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:24.945309 23282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:24.946455 23282 hybrid_clock.cc:648] HybridClock initialized: now 1786372524946419 us; error 46 us; skew 500 ppm
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:24.949343 23282 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:24.950258 23282 webserver.cc:460] Webserver started at http://127.21.43.67:45449/ using document root <none> and password file <none>
I20260810 14:35:24.950788 23282 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:24.950846 23282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:24.951011 23282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:24.953090 23282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "5d877bd039b1443c9594933fbfb167c9"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "b5074f0652a7de740d2398ec323b3f38"
server_key_iv: "b76c9d360849894f4df7b5bf38e8d3b7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.953595 23282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "5d877bd039b1443c9594933fbfb167c9"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "b5074f0652a7de740d2398ec323b3f38"
server_key_iv: "b76c9d360849894f4df7b5bf38e8d3b7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.957520 23282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:24.960167 23298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.961465 23282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:24.961611 23282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "5d877bd039b1443c9594933fbfb167c9"
format_stamp: "Formatted at 2026-08-10 14:35:24 on dist-test-slave-chsk"
server_key: "b5074f0652a7de740d2398ec323b3f38"
server_key_iv: "b76c9d360849894f4df7b5bf38e8d3b7"
server_key_version: "encryptionkey@0"
I20260810 14:35:24.961750 23282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:24.984334 23282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:24.987823 23282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:24.988058 23282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:24.989670 23282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:24.989751 23282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:24.989828 23282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:24.989879 23282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:24 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:25.002211 23282 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:45571
I20260810 14:35:25.002316 23420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:45571 every 8 connection(s)
I20260810 14:35:25.003520 23282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:25.005757 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.990750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:53859 (local address 127.21.43.126:38833)
0810 14:35:24.990885 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:24.990889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:24.992174 (+ 1285us) server_negotiation.cc:407] Connection header received
0810 14:35:24.993517 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:24.993525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:24.993600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:24.993714 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:24.995421 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.996326 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.998092 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.998318 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.998440 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.001586 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.001592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.001594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.001631 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.003402 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.004020 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.004024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.004026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.004154 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.004620 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.004624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.004625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.004899 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.004991 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.005500 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.005629 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:35:25.006556 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:24.991401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.67:53859)
0810 14:35:24.991971 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:24.992017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:24.993230 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:24.993948 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:24.993959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:24.994422 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:24.995244 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.995269 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.996496 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.996500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.997946 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:24.997956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:24.998492 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:24.998497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:24.998687 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:24.999367 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:24.999387 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:25.001364 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:25.003536 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.003541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.003553 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.003871 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.004291 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.004294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.004296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.004480 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.005007 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:25.005013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:25.005262 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.005416 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.005677 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":138,"threads_started":1}
I20260810 14:35:25.007836 23421 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38833
I20260810 14:35:25.007997 23421 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:25.008635 23421 heartbeater.cc:511] Master 127.21.43.126:38833 requested a full tablet report, sending...
I20260810 14:35:25.009951 22902 ts_manager.cc:194] Registered new tserver with Master: 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571)
I20260810 14:35:25.010706 22902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:53859
I20260810 14:35:25.010969 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23282
I20260810 14:35:25.011117 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:25.011477 21677 external_mini_cluster.cc:1428] Setting key 9f2d652c788df45e2709b2c618111512
I20260810 14:35:25.013713 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:25.018936 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.010348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.21.43.67:43145)
0810 14:35:25.010799 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.010814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.010933 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.011376 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.011379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.011649 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:25.011800 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.011811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.012762 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.012765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.013609 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.013622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.014162 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.014166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.014265 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.014856 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:25.014867 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:25.015297 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:25.017273 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.017279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.017281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.017520 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.017862 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.017866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.017867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.017959 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.018465 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:25.018467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:25.018555 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.018641 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.018776 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260810 14:35:25.019181 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.010436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:43145 (local address 127.21.43.126:38833)
0810 14:35:25.010593 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.010598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.010904 (+ 306us) server_negotiation.cc:407] Connection header received
0810 14:35:25.011073 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.011078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.011133 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.011219 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:25.011938 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.012629 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.013731 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.014028 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.014145 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.015443 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.015447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.015450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.015485 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.017135 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.017631 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.017638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.017642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.017743 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.018063 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.018069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.018074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.018362 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.018476 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.018730 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.018892 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 14:35:25 dist-test-slave-chsk krb5kdc[22853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:35:25 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:25.043924 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.035233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.21.43.126:38833)
0810 14:35:25.035385 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.035388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.035401 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:25.035443 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.035445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.035485 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.035593 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:25.036461 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.036984 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.037893 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.038178 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.038269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.040812 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.040815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.040817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.040844 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.042359 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.042745 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.042748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.042749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.042821 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.043067 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.043069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.043070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.043299 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.043405 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.043613 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.043707 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:25.047039 22902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:25.050088 22902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:25.065052 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.059477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.21.43.65:39407)
0810 14:35:25.060041 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.060050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.060074 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:35:25.060155 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.060160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.060369 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.060525 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.060877 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.062073 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.063566 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.064376 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.064493 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.064646 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.064810 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.064912 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":96,"threads_started":1}
I20260810 14:35:25.065081 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.059032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39407 (local address 127.0.0.1:41198)
0810 14:35:25.059906 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.059920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.060007 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.060532 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.060538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.060568 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.060748 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.060758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.062194 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.062198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.063435 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.063443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.064504 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.064507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.064591 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.064625 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.064761 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.064896 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":50,"thread_start_us":124,"threads_started":1}
I20260810 14:35:25.066111 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.058757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35713 (local address 127.0.0.1:49780)
0810 14:35:25.059633 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.059646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.059748 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.060770 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.060773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.060795 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.060945 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.060951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.061899 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.061904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.063431 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.063443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.065531 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.065535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.065698 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.065724 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.065836 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.065912 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":63,"thread_start_us":117,"threads_started":1}
I20260810 14:35:25.066390 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.058652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45571 (local address 127.0.0.1:46500)
0810 14:35:25.059500 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.059567 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.059978 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.060703 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.060706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.060723 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.060913 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.060924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.062329 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.062332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.063457 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.063467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.065565 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.065569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.065673 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.065696 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.065771 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.065855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":32,"thread_start_us":100,"threads_started":1}
I20260810 14:35:25.066767 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.059477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.21.43.67:45571)
0810 14:35:25.060253 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.060264 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.060284 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:35:25.060351 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.060356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.060517 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.060671 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.061080 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.062221 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.063624 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.064376 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.064493 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.064664 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.066002 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.066125 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":130,"threads_started":1}
I20260810 14:35:25.067124 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.058957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.21.43.66:35713)
0810 14:35:25.060293 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.060307 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.060330 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:35:25.060411 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.060414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.060611 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.060933 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.061096 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.061781 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.064716 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.065403 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.065519 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.066758 (+ 1239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.066817 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.066920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1222,"thread_start_us":107,"threads_started":1}
I20260810 14:35:25.068323 23048 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 (DEFAULT_TABLE table=test-table [id=f70a15388b2e4af78a695eb26f354cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.069413 23048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1b16543504bb9a73fcb6569ac7c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.069679 23352 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 (DEFAULT_TABLE table=test-table [id=f70a15388b2e4af78a695eb26f354cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.069676 23200 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 (DEFAULT_TABLE table=test-table [id=f70a15388b2e4af78a695eb26f354cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.070710 23352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1b16543504bb9a73fcb6569ac7c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.070704 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1b16543504bb9a73fcb6569ac7c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.077463 23446 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Bootstrap starting.
I20260810 14:35:25.079357 23447 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Bootstrap starting.
I20260810 14:35:25.079955 23446 tablet_bootstrap.cc:654] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.081007 23446 log.cc:824] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:25.081625 23448 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Bootstrap starting.
I20260810 14:35:25.081764 23447 tablet_bootstrap.cc:654] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.082679 23447 log.cc:824] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:25.083458 23446 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: No bootstrap required, opened a new log
I20260810 14:35:25.083684 23446 ts_tablet_manager.cc:1397] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:35:25.084152 23448 tablet_bootstrap.cc:654] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.084591 23447 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: No bootstrap required, opened a new log
I20260810 14:35:25.084793 23447 ts_tablet_manager.cc:1397] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:25.085057 23448 log.cc:824] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:25.087342 23448 tablet_bootstrap.cc:492] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: No bootstrap required, opened a new log
I20260810 14:35:25.087560 23448 ts_tablet_manager.cc:1397] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:35:25.088405 23447 raft_consensus.cc:348] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.088665 23447 raft_consensus.cc:374] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.088491 23446 raft_consensus.cc:348] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.088732 23447 raft_consensus.cc:729] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33201f03d3d4a4187dea597b5561d25, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.088755 23446 raft_consensus.cc:374] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.088809 23446 raft_consensus.cc:729] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0dc7e08fb8481bb76192066c68ecae, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.089197 23447 consensus_queue.cc:260] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.089254 23446 consensus_queue.cc:260] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.090337 23447 ts_tablet_manager.cc:1428] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 14:35:25.090337 23446 ts_tablet_manager.cc:1428] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:35:25.090740 23273 heartbeater.cc:503] Master 127.21.43.126:38833 was elected leader, sending a full tablet report...
I20260810 14:35:25.091181 23119 heartbeater.cc:503] Master 127.21.43.126:38833 was elected leader, sending a full tablet report...
I20260810 14:35:25.091687 23448 raft_consensus.cc:348] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.092010 23448 raft_consensus.cc:374] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.092082 23448 raft_consensus.cc:729] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d877bd039b1443c9594933fbfb167c9, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.092562 23448 consensus_queue.cc:260] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.093667 23448 ts_tablet_manager.cc:1428] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:35:25.093849 23421 heartbeater.cc:503] Master 127.21.43.126:38833 was elected leader, sending a full tablet report...
W20260810 14:35:25.254837 23422 tablet.cc:2369] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:25.263625 23274 tablet.cc:2369] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:25.292436 23120 tablet.cc:2369] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:25.316409 23454 raft_consensus.cc:482] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:25.316708 23454 raft_consensus.cc:504] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.317987 23454 leader_election.cc:288] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571), d33201f03d3d4a4187dea597b5561d25 (127.21.43.66:35713)
I20260810 14:35:25.322134 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.318462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:39101 (local address 127.21.43.67:45571)
0810 14:35:25.318604 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.318608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.318791 (+ 183us) server_negotiation.cc:407] Connection header received
0810 14:35:25.318967 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.318973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.319022 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.319110 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.319431 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.319966 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321129 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.321524 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321668 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.321726 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.321908 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.321976 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:35:25.322211 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.318441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:40769 (local address 127.21.43.66:35713)
0810 14:35:25.318594 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.318598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.318999 (+ 401us) server_negotiation.cc:407] Connection header received
0810 14:35:25.319191 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.319197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.319248 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.319338 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.319615 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.320110 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321181 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.321581 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321722 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.321779 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.321994 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.322068 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:25.322190 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.318314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45571 (local address 127.21.43.65:39101)
0810 14:35:25.318684 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.318697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.318845 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.319133 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.319136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.319182 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.319316 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.319321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.320095 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.320099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.320989 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.320999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.321666 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.321767 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.321791 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.321892 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.321970 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
I20260810 14:35:25.322233 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.318314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35713 (local address 127.21.43.65:40769)
0810 14:35:25.318896 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.318909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.319037 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.319350 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.319352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.319369 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.319494 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.319500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.320259 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.320262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.321036 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.321045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.321727 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.321731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.321856 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.321874 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.321965 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.322053 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":306,"thread_start_us":106,"threads_started":1}
I20260810 14:35:25.323288 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1b16543504bb9a73fcb6569ac7c18" candidate_uuid: "df0dc7e08fb8481bb76192066c68ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9" is_pre_election: true
I20260810 14:35:25.323277 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1b16543504bb9a73fcb6569ac7c18" candidate_uuid: "df0dc7e08fb8481bb76192066c68ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33201f03d3d4a4187dea597b5561d25" is_pre_election: true
I20260810 14:35:25.323634 23223 raft_consensus.cc:2454] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0dc7e08fb8481bb76192066c68ecae in term 0.
I20260810 14:35:25.323634 23374 raft_consensus.cc:2454] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0dc7e08fb8481bb76192066c68ecae in term 0.
I20260810 14:35:25.324177 22997 leader_election.cc:302] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d877bd039b1443c9594933fbfb167c9, df0dc7e08fb8481bb76192066c68ecae; no voters:
I20260810 14:35:25.324442 23454 raft_consensus.cc:2781] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:25.324532 23454 raft_consensus.cc:482] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:25.324566 23454 raft_consensus.cc:3037] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.325726 23454 raft_consensus.cc:504] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.326149 23454 leader_election.cc:288] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [CANDIDATE]: Term 1 election: Requested vote from peers 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571), d33201f03d3d4a4187dea597b5561d25 (127.21.43.66:35713)
I20260810 14:35:25.326562 23223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1b16543504bb9a73fcb6569ac7c18" candidate_uuid: "df0dc7e08fb8481bb76192066c68ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33201f03d3d4a4187dea597b5561d25"
I20260810 14:35:25.326574 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1b16543504bb9a73fcb6569ac7c18" candidate_uuid: "df0dc7e08fb8481bb76192066c68ecae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9"
I20260810 14:35:25.326720 23223 raft_consensus.cc:3037] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.326720 23374 raft_consensus.cc:3037] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.328306 23223 raft_consensus.cc:2454] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0dc7e08fb8481bb76192066c68ecae in term 1.
I20260810 14:35:25.328514 23374 raft_consensus.cc:2454] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0dc7e08fb8481bb76192066c68ecae in term 1.
I20260810 14:35:25.328797 22999 leader_election.cc:302] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d33201f03d3d4a4187dea597b5561d25, df0dc7e08fb8481bb76192066c68ecae; no voters:
I20260810 14:35:25.329000 23454 raft_consensus.cc:2781] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:25.329262 23454 raft_consensus.cc:686] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 1 LEADER]: Becoming Leader. State: Replica: df0dc7e08fb8481bb76192066c68ecae, State: Running, Role: LEADER
I20260810 14:35:25.329679 23454 consensus_queue.cc:237] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } }
I20260810 14:35:25.333263 22903 catalog_manager.cc:5495] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae reported cstate change: term changed from 0 to 1, leader changed from <none> to df0dc7e08fb8481bb76192066c68ecae (127.21.43.65). New cstate: current_term: 1 leader_uuid: "df0dc7e08fb8481bb76192066c68ecae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } health_report { overall_health: HEALTHY } } }
I20260810 14:35:25.371297 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.367199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.21.43.65:39407)
0810 14:35:25.367354 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.367361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.367378 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:35:25.367645 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.367648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.367709 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.367820 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:25.368110 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.368631 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.369562 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.369786 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.369902 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.370275 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.370806 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.370949 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.371034 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.371104 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:35:25.378309 23223 raft_consensus.cc:1261] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Refusing update from remote peer df0dc7e08fb8481bb76192066c68ecae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.379340 23374 raft_consensus.cc:1261] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Refusing update from remote peer df0dc7e08fb8481bb76192066c68ecae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.379299 23464 consensus_queue.cc:1035] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.379984 23464 consensus_queue.cc:1035] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.388911 23468 mvcc.cc:205] Tried to move back new op lower bound from 7316981863940046848 to 7316981863753420800. Current Snapshot: MvccSnapshot[applied={T|T < 7316981863940046848}]
I20260810 14:35:25.389119 23470 mvcc.cc:205] Tried to move back new op lower bound from 7316981863940046848 to 7316981863753420800. Current Snapshot: MvccSnapshot[applied={T|T < 7316981863940046848}]
I20260810 14:35:25.402103 22901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:25.402555 22901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:25.405651 22901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Sending DeleteTablet for 3 replicas of tablet 8ea1b16543504bb9a73fcb6569ac7c18
I20260810 14:35:25.406764 23200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:49780
I20260810 14:35:25.406755 23352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:46500
I20260810 14:35:25.407374 23048 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1b16543504bb9a73fcb6569ac7c18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41198
I20260810 14:35:25.407544 23478 tablet_replica.cc:330] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: stopping tablet replica
I20260810 14:35:25.408099 23478 raft_consensus.cc:2229] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:25.408165 23479 tablet_replica.cc:330] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: stopping tablet replica
I20260810 14:35:25.408156 22901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:35:25.408492 23479 raft_consensus.cc:2229] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 1 LEADER]: Raft consensus shutting down.
W20260810 14:35:25.408667 22901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:25.408710 23478 raft_consensus.cc:2258] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.408943 23479 raft_consensus.cc:2258] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.409835 23480 tablet_replica.cc:330] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: stopping tablet replica
I20260810 14:35:25.410156 23480 raft_consensus.cc:2229] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:25.410483 23480 raft_consensus.cc:2258] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.411412 23478 ts_tablet_manager.cc:1905] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.412057 23480 ts_tablet_manager.cc:1905] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.414690 23478 ts_tablet_manager.cc:1918] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:25.414827 23480 ts_tablet_manager.cc:1918] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:25.414942 23480 log.cc:1196] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/8ea1b16543504bb9a73fcb6569ac7c18
I20260810 14:35:25.415040 23478 log.cc:1196] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/8ea1b16543504bb9a73fcb6569ac7c18
I20260810 14:35:25.415303 23480 ts_tablet_manager.cc:1939] T 8ea1b16543504bb9a73fcb6569ac7c18 P d33201f03d3d4a4187dea597b5561d25: Deleting consensus metadata
I20260810 14:35:25.415611 23478 ts_tablet_manager.cc:1939] T 8ea1b16543504bb9a73fcb6569ac7c18 P 5d877bd039b1443c9594933fbfb167c9: Deleting consensus metadata
I20260810 14:35:25.416723 22890 catalog_manager.cc:4841] TS d33201f03d3d4a4187dea597b5561d25 (127.21.43.66:35713): tablet 8ea1b16543504bb9a73fcb6569ac7c18 (table test-table [id=f70a15388b2e4af78a695eb26f354cce]) successfully deleted
I20260810 14:35:25.416967 22888 catalog_manager.cc:4841] TS 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571): tablet 8ea1b16543504bb9a73fcb6569ac7c18 (table test-table [id=f70a15388b2e4af78a695eb26f354cce]) successfully deleted
I20260810 14:35:25.417702 23048 tablet_service.cc:1467] Processing CreateTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 (DEFAULT_TABLE table=test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.418058 23048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e28c97c2245f1a2cd20a5f14459e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.418018 23200 tablet_service.cc:1467] Processing CreateTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 (DEFAULT_TABLE table=test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.418264 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e28c97c2245f1a2cd20a5f14459e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.418889 23479 ts_tablet_manager.cc:1905] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.420857 23447 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Bootstrap starting.
I20260810 14:35:25.421748 23479 ts_tablet_manager.cc:1918] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:25.421936 23479 log.cc:1196] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/8ea1b16543504bb9a73fcb6569ac7c18
I20260810 14:35:25.422006 23447 tablet_bootstrap.cc:654] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.422407 23479 ts_tablet_manager.cc:1939] T 8ea1b16543504bb9a73fcb6569ac7c18 P df0dc7e08fb8481bb76192066c68ecae: Deleting consensus metadata
I20260810 14:35:25.423192 23446 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Bootstrap starting.
I20260810 14:35:25.423820 22886 catalog_manager.cc:4841] TS df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407): tablet 8ea1b16543504bb9a73fcb6569ac7c18 (table test-table [id=f70a15388b2e4af78a695eb26f354cce]) successfully deleted
I20260810 14:35:25.424252 23446 tablet_bootstrap.cc:654] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.425326 23352 tablet_service.cc:1467] Processing CreateTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 (DEFAULT_TABLE table=test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:25.425608 23446 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: No bootstrap required, opened a new log
I20260810 14:35:25.425642 23352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09e28c97c2245f1a2cd20a5f14459e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.425725 23446 ts_tablet_manager.cc:1397] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:25.426087 23447 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: No bootstrap required, opened a new log
I20260810 14:35:25.426180 23447 ts_tablet_manager.cc:1397] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 14:35:25.426285 23446 raft_consensus.cc:348] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.426406 23446 raft_consensus.cc:374] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.426440 23446 raft_consensus.cc:729] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0dc7e08fb8481bb76192066c68ecae, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.426723 23447 raft_consensus.cc:348] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.426842 23447 raft_consensus.cc:374] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.426877 23447 raft_consensus.cc:729] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33201f03d3d4a4187dea597b5561d25, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.427026 23447 consensus_queue.cc:260] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.427332 23447 ts_tablet_manager.cc:1428] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:35:25.427825 23446 consensus_queue.cc:260] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.429097 23446 ts_tablet_manager.cc:1428] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:25.429220 23448 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Bootstrap starting.
I20260810 14:35:25.430406 23448 tablet_bootstrap.cc:654] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.432215 23448 tablet_bootstrap.cc:492] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: No bootstrap required, opened a new log
I20260810 14:35:25.432332 23448 ts_tablet_manager.cc:1397] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:25.432865 23448 raft_consensus.cc:348] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.432971 23448 raft_consensus.cc:374] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.432992 23448 raft_consensus.cc:729] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d877bd039b1443c9594933fbfb167c9, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.433172 23448 consensus_queue.cc:260] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.433475 23448 ts_tablet_manager.cc:1428] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:35:25.670810 23452 raft_consensus.cc:482] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:25.671002 23452 raft_consensus.cc:504] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.672212 23452 leader_election.cc:288] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407), 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571)
I20260810 14:35:25.678249 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.672715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:58311 (local address 127.21.43.65:39407)
0810 14:35:25.672832 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.672836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.673105 (+ 269us) server_negotiation.cc:407] Connection header received
0810 14:35:25.673229 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.673232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.673285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.673387 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.673941 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.674744 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.676720 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.677358 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.677505 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.677566 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.678016 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.678111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:35:25.678249 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.672634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:44105 (local address 127.21.43.67:45571)
0810 14:35:25.672791 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.672796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.673091 (+ 295us) server_negotiation.cc:407] Connection header received
0810 14:35:25.673229 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.673232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.673285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.673387 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:25.673941 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.674754 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.676715 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.677358 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.677505 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.677566 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.678016 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.678111 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:35:25.678309 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.672546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39407 (local address 127.21.43.66:58311)
0810 14:35:25.673011 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.673025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.673115 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.673447 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.673450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.673499 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.673794 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.673800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.674903 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.674907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.676541 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.676555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.677524 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.677530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.677678 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.677834 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.677937 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.678061 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":314,"spinlock_wait_cycles":40960,"thread_start_us":105,"threads_started":1}
I20260810 14:35:25.678318 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.672546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45571 (local address 127.21.43.66:44105)
0810 14:35:25.672955 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.672973 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.673114 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.673447 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.673450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.673499 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:25.673786 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.673796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.674903 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.674908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.676541 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.676555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.677537 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.677540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.677818 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.677839 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.677934 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.678061 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":353,"spinlock_wait_cycles":7936,"thread_start_us":95,"threads_started":1}
I20260810 14:35:25.678951 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e28c97c2245f1a2cd20a5f14459e5" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0dc7e08fb8481bb76192066c68ecae" is_pre_election: true
I20260810 14:35:25.679127 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e28c97c2245f1a2cd20a5f14459e5" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9" is_pre_election: true
I20260810 14:35:25.679258 23057 raft_consensus.cc:2454] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 0.
I20260810 14:35:25.679333 23374 raft_consensus.cc:2454] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 0.
I20260810 14:35:25.679824 23146 leader_election.cc:302] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d877bd039b1443c9594933fbfb167c9, d33201f03d3d4a4187dea597b5561d25, df0dc7e08fb8481bb76192066c68ecae; no voters:
I20260810 14:35:25.680112 23452 raft_consensus.cc:2781] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:25.680179 23452 raft_consensus.cc:482] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:25.680236 23452 raft_consensus.cc:3037] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.681279 23452 raft_consensus.cc:504] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.681708 23452 leader_election.cc:288] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 election: Requested vote from peers df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407), 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571)
I20260810 14:35:25.682153 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e28c97c2245f1a2cd20a5f14459e5" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0dc7e08fb8481bb76192066c68ecae"
I20260810 14:35:25.682217 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09e28c97c2245f1a2cd20a5f14459e5" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9"
I20260810 14:35:25.682299 23057 raft_consensus.cc:3037] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.682394 23374 raft_consensus.cc:3037] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.683439 23057 raft_consensus.cc:2454] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 1.
I20260810 14:35:25.683468 23374 raft_consensus.cc:2454] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 1.
I20260810 14:35:25.683787 23146 leader_election.cc:302] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d877bd039b1443c9594933fbfb167c9, d33201f03d3d4a4187dea597b5561d25, df0dc7e08fb8481bb76192066c68ecae; no voters:
I20260810 14:35:25.684013 23452 raft_consensus.cc:2781] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:25.684262 23452 raft_consensus.cc:686] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 1 LEADER]: Becoming Leader. State: Replica: d33201f03d3d4a4187dea597b5561d25, State: Running, Role: LEADER
I20260810 14:35:25.684536 23452 consensus_queue.cc:237] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.687197 22901 catalog_manager.cc:5495] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d33201f03d3d4a4187dea597b5561d25 (127.21.43.66). New cstate: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:35:25 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:25.744694 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.733841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38833 (local address 127.0.0.1:36552)
0810 14:35:25.734661 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.734678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.734836 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.735453 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.735455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.735871 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:25.735995 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.736001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.736868 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.736871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.737518 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.737527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.738144 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.738147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.738224 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.738816 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:25.738834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:25.740720 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:25.742781 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.742791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.742798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.743094 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.743511 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:25.743514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:25.743516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:25.743601 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:25.744019 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:25.744028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:25.744147 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.744270 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.744466 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"spinlock_wait_cycles":2944,"thread_start_us":135,"threads_started":1}
I20260810 14:35:25.744695 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.733821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.21.43.126:38833)
0810 14:35:25.734181 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.734193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.734817 (+ 624us) server_negotiation.cc:407] Connection header received
0810 14:35:25.735257 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.735260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.735308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.735397 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:25.736106 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.736753 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.737655 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.737995 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.738113 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.740887 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.740891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.740893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.740915 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.742621 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.743303 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.743307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.743310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.743401 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.743695 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.743700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.743701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.743901 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.744021 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.744328 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.744539 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":276,"thread_start_us":172,"threads_started":1}
I20260810 14:35:25.749414 22901 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:36552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:35:25.750260 22901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:25.755831 23048 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07915414fd4ae891b799293c6f5b60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bb2a1914a27471b8a8f93727abc1a31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:25.755945 23200 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07915414fd4ae891b799293c6f5b60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bb2a1914a27471b8a8f93727abc1a31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:25.756069 23352 tablet_service.cc:1467] Processing CreateTablet for tablet 9f07915414fd4ae891b799293c6f5b60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bb2a1914a27471b8a8f93727abc1a31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:25.756322 23048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07915414fd4ae891b799293c6f5b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.756361 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07915414fd4ae891b799293c6f5b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.756415 23352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f07915414fd4ae891b799293c6f5b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:25.758845 23447 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: Bootstrap starting.
I20260810 14:35:25.759337 23446 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: Bootstrap starting.
I20260810 14:35:25.760001 23447 tablet_bootstrap.cc:654] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.760309 23446 tablet_bootstrap.cc:654] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.760488 23448 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: Bootstrap starting.
I20260810 14:35:25.761523 23448 tablet_bootstrap.cc:654] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:25.761735 23446 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: No bootstrap required, opened a new log
I20260810 14:35:25.761827 23446 ts_tablet_manager.cc:1397] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:25.762194 23447 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: No bootstrap required, opened a new log
I20260810 14:35:25.762295 23447 ts_tablet_manager.cc:1397] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:25.762293 23446 raft_consensus.cc:348] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.762419 23446 raft_consensus.cc:374] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.762452 23446 raft_consensus.cc:729] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0dc7e08fb8481bb76192066c68ecae, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.762574 23446 consensus_queue.cc:260] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.762786 23447 raft_consensus.cc:348] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.762835 23454 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.762903 23447 raft_consensus.cc:374] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.762936 23447 raft_consensus.cc:729] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33201f03d3d4a4187dea597b5561d25, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.762959 23454 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.763054 23447 consensus_queue.cc:260] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.763240 23446 ts_tablet_manager.cc:1428] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:35:25.763415 23447 ts_tablet_manager.cc:1428] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:35:25.763700 23452 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.763904 23452 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.764132 23448 tablet_bootstrap.cc:492] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: No bootstrap required, opened a new log
I20260810 14:35:25.764227 23448 ts_tablet_manager.cc:1397] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:25.764658 23448 raft_consensus.cc:348] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.764787 23448 raft_consensus.cc:374] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:25.764815 23448 raft_consensus.cc:729] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d877bd039b1443c9594933fbfb167c9, State: Initialized, Role: FOLLOWER
I20260810 14:35:25.764919 23448 consensus_queue.cc:260] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.765158 23448 ts_tablet_manager.cc:1428] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:25.766839 23455 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.767009 23455 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.816290 23452 raft_consensus.cc:482] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:25.816439 23452 raft_consensus.cc:504] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.817030 23452 leader_election.cc:288] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407), 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571)
I20260810 14:35:25.817623 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07915414fd4ae891b799293c6f5b60" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9" is_pre_election: true
I20260810 14:35:25.817809 23374 raft_consensus.cc:2454] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 0.
I20260810 14:35:25.818136 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07915414fd4ae891b799293c6f5b60" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0dc7e08fb8481bb76192066c68ecae" is_pre_election: true
I20260810 14:35:25.818197 23146 leader_election.cc:302] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d877bd039b1443c9594933fbfb167c9, d33201f03d3d4a4187dea597b5561d25; no voters:
I20260810 14:35:25.818305 23057 raft_consensus.cc:2454] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 0.
I20260810 14:35:25.818478 23452 raft_consensus.cc:2781] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:25.818568 23452 raft_consensus.cc:482] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:25.818625 23452 raft_consensus.cc:3037] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.819819 23452 raft_consensus.cc:504] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.820214 23452 leader_election.cc:288] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 election: Requested vote from peers df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407), 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571)
I20260810 14:35:25.820672 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07915414fd4ae891b799293c6f5b60" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0dc7e08fb8481bb76192066c68ecae"
I20260810 14:35:25.820685 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f07915414fd4ae891b799293c6f5b60" candidate_uuid: "d33201f03d3d4a4187dea597b5561d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d877bd039b1443c9594933fbfb167c9"
I20260810 14:35:25.820801 23057 raft_consensus.cc:3037] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.820830 23374 raft_consensus.cc:3037] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:25.821820 23057 raft_consensus.cc:2454] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 1.
I20260810 14:35:25.821848 23374 raft_consensus.cc:2454] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33201f03d3d4a4187dea597b5561d25 in term 1.
I20260810 14:35:25.822175 23145 leader_election.cc:302] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d33201f03d3d4a4187dea597b5561d25, df0dc7e08fb8481bb76192066c68ecae; no voters:
I20260810 14:35:25.822429 23452 raft_consensus.cc:2781] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:25.822531 23452 raft_consensus.cc:686] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [term 1 LEADER]: Becoming Leader. State: Replica: d33201f03d3d4a4187dea597b5561d25, State: Running, Role: LEADER
I20260810 14:35:25.822674 23452 consensus_queue.cc:237] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } }
I20260810 14:35:25.823490 23486 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: TxnStatusTablet state changed. Reason: New leader d33201f03d3d4a4187dea597b5561d25. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.823621 23486 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:25.823969 23501 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:35:25.824169 23501 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:35:25.824752 22903 catalog_manager.cc:5495] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d33201f03d3d4a4187dea597b5561d25 (127.21.43.66). New cstate: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:25.852411 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.848036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.21.43.66:35713)
0810 14:35:25.848328 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.848335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.848348 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:25.848417 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.848419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.848480 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.848582 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:25.848885 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.849461 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.850717 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.850960 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.851046 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.851463 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.851960 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.852072 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.852190 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.852251 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
I20260810 14:35:25.852409 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.847924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35713 (local address 127.0.0.1:49796)
0810 14:35:25.848076 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.848092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.848198 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.848626 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.848629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.848638 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:25.848760 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.848766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.849583 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.849586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.850572 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.850584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.851085 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.851089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.851218 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.851280 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.852074 (+ 794us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.852092 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.852167 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.852261 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:35:25.855686 23374 raft_consensus.cc:1261] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Refusing update from remote peer d33201f03d3d4a4187dea597b5561d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.855901 23057 raft_consensus.cc:1261] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Refusing update from remote peer d33201f03d3d4a4187dea597b5561d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.856346 23486 consensus_queue.cc:1035] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.856545 23452 consensus_queue.cc:1035] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.858048 23455 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: TxnStatusTablet state changed. Reason: New leader d33201f03d3d4a4187dea597b5561d25. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.858176 23455 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.859828 23454 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: TxnStatusTablet state changed. Reason: New leader d33201f03d3d4a4187dea597b5561d25. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.859974 23454 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.861380 23452 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.861508 23452 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:25.861593 23455 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.861649 23452 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.861706 23455 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P 5d877bd039b1443c9594933fbfb167c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.861718 23452 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P d33201f03d3d4a4187dea597b5561d25: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:25.865546 23467 mvcc.cc:205] Tried to move back new op lower bound from 7316981865899458560 to 7316981865771479040. Current Snapshot: MvccSnapshot[applied={T|T < 7316981865899458560 or (T in {7316981865899458560})}]
I20260810 14:35:25.865800 23454 tablet_replica.cc:439] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d33201f03d3d4a4187dea597b5561d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 } } peers { permanent_uuid: "d33201f03d3d4a4187dea597b5561d25" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35713 } } peers { permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 } } }
I20260810 14:35:25.865921 23454 tablet_replica.cc:442] T 9f07915414fd4ae891b799293c6f5b60 P df0dc7e08fb8481bb76192066c68ecae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:25.872031 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.867503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.21.43.66:35713)
0810 14:35:25.867651 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.867656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.867770 (+ 114us) server_negotiation.cc:407] Connection header received
0810 14:35:25.867888 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.867892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.867940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.868007 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:25.868394 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.870005 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.870880 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.871120 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.871276 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.871561 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.871645 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.871730 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.871825 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.871884 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:35:25.884967 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.880468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:46141 (local address 127.21.43.66:35713)
0810 14:35:25.880641 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.880646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.880845 (+ 199us) server_negotiation.cc:407] Connection header received
0810 14:35:25.880991 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.880993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.881037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.881118 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:25.881496 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.882031 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.883705 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.883922 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.884018 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.884462 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.884556 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.884628 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.884785 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.884840 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":79}
I20260810 14:35:25.885026 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.880309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35713 (local address 127.21.43.66:46141)
0810 14:35:25.880741 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:25.880758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:25.880882 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:25.881153 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:25.881156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:25.881163 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:25.881343 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.881350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.882239 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.882243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.883551 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:25.883565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.884074 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.884078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:25.884209 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.884277 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:25.884656 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:25.884663 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:35:25.884734 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:35:25.884823 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":19,"thread_start_us":210,"threads_started":1}
I20260810 14:35:25.894551 23057 raft_consensus.cc:1261] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Refusing update from remote peer d33201f03d3d4a4187dea597b5561d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.894620 23374 raft_consensus.cc:1261] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Refusing update from remote peer d33201f03d3d4a4187dea597b5561d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:25.895012 23452 consensus_queue.cc:1035] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0dc7e08fb8481bb76192066c68ecae" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.895263 23486 consensus_queue.cc:1035] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d877bd039b1443c9594933fbfb167c9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:25.943252 22901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:25.943449 22901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:25.945832 22901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7be47b7994754ae7bf3435ec520f0ae0: Sending DeleteTablet for 3 replicas of tablet d09e28c97c2245f1a2cd20a5f14459e5
I20260810 14:35:25.946560 23048 tablet_service.cc:1514] Processing DeleteTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41198
I20260810 14:35:25.946597 23199 tablet_service.cc:1514] Processing DeleteTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:49780
I20260810 14:35:25.946957 23517 tablet_replica.cc:330] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: stopping tablet replica
I20260810 14:35:25.946945 23352 tablet_service.cc:1514] Processing DeleteTablet for tablet d09e28c97c2245f1a2cd20a5f14459e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:25 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:46500
I20260810 14:35:25.947106 23517 raft_consensus.cc:2229] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:25.947320 23519 tablet_replica.cc:330] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: stopping tablet replica
I20260810 14:35:25.947404 23517 raft_consensus.cc:2258] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.947544 23519 raft_consensus.cc:2229] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:25.947731 23519 raft_consensus.cc:2258] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.948355 23517 ts_tablet_manager.cc:1905] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.948494 23519 ts_tablet_manager.cc:1905] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.948757 23518 tablet_replica.cc:330] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: stopping tablet replica
I20260810 14:35:25.948964 23518 raft_consensus.cc:2229] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:25.949363 23518 raft_consensus.cc:2258] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:25.950177 23518 ts_tablet_manager.cc:1905] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:25.951521 23517 ts_tablet_manager.cc:1918] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:25.951521 23519 ts_tablet_manager.cc:1918] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:25.951608 23519 log.cc:1196] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/d09e28c97c2245f1a2cd20a5f14459e5
I20260810 14:35:25.951737 23517 log.cc:1196] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/d09e28c97c2245f1a2cd20a5f14459e5
I20260810 14:35:25.951947 23519 ts_tablet_manager.cc:1939] T d09e28c97c2245f1a2cd20a5f14459e5 P 5d877bd039b1443c9594933fbfb167c9: Deleting consensus metadata
I20260810 14:35:25.952086 23517 ts_tablet_manager.cc:1939] T d09e28c97c2245f1a2cd20a5f14459e5 P df0dc7e08fb8481bb76192066c68ecae: Deleting consensus metadata
I20260810 14:35:25.952804 22886 catalog_manager.cc:4841] TS df0dc7e08fb8481bb76192066c68ecae (127.21.43.65:39407): tablet d09e28c97c2245f1a2cd20a5f14459e5 (table test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]) successfully deleted
I20260810 14:35:25.952804 22888 catalog_manager.cc:4841] TS 5d877bd039b1443c9594933fbfb167c9 (127.21.43.67:45571): tablet d09e28c97c2245f1a2cd20a5f14459e5 (table test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]) successfully deleted
I20260810 14:35:25.953289 23518 ts_tablet_manager.cc:1918] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:25.953387 23518 log.cc:1196] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/d09e28c97c2245f1a2cd20a5f14459e5
I20260810 14:35:25.953688 23518 ts_tablet_manager.cc:1939] T d09e28c97c2245f1a2cd20a5f14459e5 P d33201f03d3d4a4187dea597b5561d25: Deleting consensus metadata
I20260810 14:35:25.954432 22890 catalog_manager.cc:4841] TS d33201f03d3d4a4187dea597b5561d25 (127.21.43.66:35713): tablet d09e28c97c2245f1a2cd20a5f14459e5 (table test-table [id=c75f0c9b8e724cbf9f12d184ec450fa8]) successfully deleted
Aug 10 14:35:25 dist-test-slave-chsk krb5kdc[22853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:25.965680 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.954693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.21.43.65:39407)
0810 14:35:25.954833 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.954836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.955181 (+ 345us) server_negotiation.cc:407] Connection header received
0810 14:35:25.955342 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.955345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.955397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.955488 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:25.956170 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.956694 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.957558 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.957800 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.957911 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.961450 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.961461 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.961475 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.961503 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.963597 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.964112 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.964118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.964119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.964206 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.964509 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.964512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.964516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.964798 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.965039 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.965302 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.965445 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:35:25.966281 23112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41212
I20260810 14:35:25.978585 23494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:25.968754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.21.43.126:38833)
0810 14:35:25.969065 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:25.969071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:25.970093 (+ 1022us) server_negotiation.cc:407] Connection header received
0810 14:35:25.970164 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:25.970167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:25.970223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:25.970340 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:25.971213 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.972065 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.972912 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:25.973261 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:25.973387 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:25.974995 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:25.974999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:25.975002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:25.975036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:25.976718 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.977152 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.977157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.977158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.977261 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:25.977571 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:25.977575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:25.977577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:25.977842 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:25.977988 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:25.978243 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 14:35:25.978400 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
W20260810 14:35:25.979110 22901 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36562
I20260810 14:35:25.980736 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22979
I20260810 14:35:25.991966 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23128
I20260810 14:35:26.003036 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23282
I20260810 14:35:26.014020 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 22869
2026-08-10T14:35:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3249 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:26 dist-test-slave-chsk krb5kdc[23526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:26 dist-test-slave-chsk krb5kdc[23526](info): set up 2 sockets
Aug 10 14:35:26 dist-test-slave-chsk krb5kdc[23526](info): commencing operation
krb5kdc: starting...
W20260810 14:35:27.177410 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 14:35:27.411201 21677 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.234s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 14:35:27.549678 21677 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.138s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 14:35:27.731700 21677 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.182s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/test-user.keytab.
W20260810 14:35:28.231514 21677 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.500s user 0.001s sys 0.001s
Aug 10 14:35:28 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:28.294677 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41847
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:41847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:28.420871 23539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:28.421366 23539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:28.421473 23539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:28.425841 23539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:28.425922 23539 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:35:28.425940 23539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:28.425961 23539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:28.425976 23539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:28.426024 23539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:28.432433 23539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:41847
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:28.433763 23539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:28.434914 23539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:28.445150 23545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.445215 23539 server_base.cc:1034] running on GCE node
W20260810 14:35:28.445187 23544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.445163 23547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.446112 23539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:28.447444 23539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:28.448652 23539 hybrid_clock.cc:648] HybridClock initialized: now 1786372528448613 us; error 56 us; skew 500 ppm
Aug 10 14:35:28 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:28.451844 23539 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:28.452875 23539 webserver.cc:460] Webserver started at http://127.21.43.126:38917/ using document root <none> and password file <none>
I20260810 14:35:28.453465 23539 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:28.453536 23539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:28.453724 23539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:28.455971 23539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "2b048d9d54d4408ebe2732f41d0f01a2"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "8721a2bcffb2647eb2b6d1d5fa4891cf"
server_key_iv: "beb0d314231ea9fbecb489b038a67d4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.456612 23539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "2b048d9d54d4408ebe2732f41d0f01a2"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "8721a2bcffb2647eb2b6d1d5fa4891cf"
server_key_iv: "beb0d314231ea9fbecb489b038a67d4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.460666 23539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:28.463303 23555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.464921 23539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:28.465057 23539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "2b048d9d54d4408ebe2732f41d0f01a2"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "8721a2bcffb2647eb2b6d1d5fa4891cf"
server_key_iv: "beb0d314231ea9fbecb489b038a67d4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.465173 23539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:28.477705 23539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:28.481575 23539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:28.481767 23539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:28.490851 23539 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:41847
I20260810 14:35:28.490863 23625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:41847 every 8 connection(s)
I20260810 14:35:28.492025 23539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:28.494251 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23539
I20260810 14:35:28.494359 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:28.494668 21677 external_mini_cluster.cc:1428] Setting key ad0b8896d5984e54989cfbffd062bbe5
I20260810 14:35:28.495549 23626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:28.503306 23626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Bootstrap starting.
Aug 10 14:35:28 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:28.506067 23626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:28.507629 23626 log.cc:824] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:28.510067 23626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: No bootstrap required, opened a new log
I20260810 14:35:28.511936 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.496319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.21.43.126:41847)
0810 14:35:28.496822 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:28.496844 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:28.496889 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:35:28.497802 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:28.497827 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:28.498287 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:28.498696 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:28.499597 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.501097 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.502143 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.502862 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.503033 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.506103 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:28.506111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:28.506131 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:28.506179 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:28.508551 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.509095 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.509101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.509108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.509241 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.509558 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.509562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.509564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.509994 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:28.510225 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:28.510550 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 14:35:28.510876 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":143,"threads_started":1}
I20260810 14:35:28.514072 23626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } }
I20260810 14:35:28.514345 23626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:28.514436 23626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b048d9d54d4408ebe2732f41d0f01a2, State: Initialized, Role: FOLLOWER
I20260810 14:35:28.514926 23626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } }
I20260810 14:35:28.515067 23626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:28.515121 23626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:28.515267 23626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:28.516314 23626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } }
I20260810 14:35:28.516651 23626 leader_election.cc:302] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b048d9d54d4408ebe2732f41d0f01a2; no voters:
I20260810 14:35:28.516950 23626 leader_election.cc:288] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:28.517221 23632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:28.517550 23632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b048d9d54d4408ebe2732f41d0f01a2, State: Running, Role: LEADER
I20260810 14:35:28.517989 23632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } }
I20260810 14:35:28.518381 23626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:28.520058 23634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b048d9d54d4408ebe2732f41d0f01a2. Latest consensus state: current_term: 1 leader_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } } }
I20260810 14:35:28.520059 23633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b048d9d54d4408ebe2732f41d0f01a2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41847 } } }
I20260810 14:35:28.520241 23634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:28.520256 23633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:28.520690 23646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:28.524637 23646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:28.531004 23646 catalog_manager.cc:1347] Generated new cluster ID: 9ac5d42b3cc5492391efe1dc4da90f6a
I20260810 14:35:28.531105 23646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:28.557040 23646 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:28.558092 23646 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:28.566352 23646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Generated new TSK 0
I20260810 14:35:28.567366 23646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:28.610263 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41847
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:28.732772 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:28.733070 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:28.733120 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:28.737301 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:28.737378 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:28.737473 23653 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:35:28.737501 23653 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:28.737564 23653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:28.743000 23653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:41847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:28.744107 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:28.745126 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:28.755481 23659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.755511 23661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.755577 23653 server_base.cc:1034] running on GCE node
W20260810 14:35:28.755517 23663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.756237 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:28.756942 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:28.758153 23653 hybrid_clock.cc:648] HybridClock initialized: now 1786372528758133 us; error 29 us; skew 500 ppm
Aug 10 14:35:28 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:28.761317 23653 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:28.762343 23653 webserver.cc:460] Webserver started at http://127.21.43.65:40121/ using document root <none> and password file <none>
I20260810 14:35:28.762933 23653 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:28.763031 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:28.763334 23653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:28.765424 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "bce61a92bba04aab86519939b6cdf59a"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "34ad22bbc2c0196a1878e55c35fac6ea"
server_key_iv: "179730af36fbf919ff9e9ee044a5e858"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.766072 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "bce61a92bba04aab86519939b6cdf59a"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "34ad22bbc2c0196a1878e55c35fac6ea"
server_key_iv: "179730af36fbf919ff9e9ee044a5e858"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.770207 23653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:28.772992 23671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.774330 23653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:28.774489 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "bce61a92bba04aab86519939b6cdf59a"
format_stamp: "Formatted at 2026-08-10 14:35:28 on dist-test-slave-chsk"
server_key: "34ad22bbc2c0196a1878e55c35fac6ea"
server_key_iv: "179730af36fbf919ff9e9ee044a5e858"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.774633 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:28.795724 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:28.800707 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:28.800992 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:28.802748 23653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:28.802830 23653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.802907 23653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:28.802953 23653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:28 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:28.820840 23653 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39703
I20260810 14:35:28.820922 23793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39703 every 8 connection(s)
I20260810 14:35:28.823320 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:28.826383 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.803723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:51899 (local address 127.21.43.126:41847)
0810 14:35:28.803864 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:28.803868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:28.805015 (+ 1147us) server_negotiation.cc:407] Connection header received
0810 14:35:28.806119 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:28.806125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:28.806200 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:28.806312 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:28.807740 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.808439 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.810406 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.810762 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.810986 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.817939 (+ 6953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:28.817965 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:28.817969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:28.818061 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:28.821802 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.822975 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.822993 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.822997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.823226 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.824011 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.824019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.824034 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.824519 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:28.824782 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:28.825840 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0810 14:35:28.826128 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:35:28.828053 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.804303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.65:51899)
0810 14:35:28.804837 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:28.804882 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:28.805835 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:28.806552 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:28.806559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:28.806917 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:28.807492 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.807514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.808585 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.808588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.810156 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.810172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.811179 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.811188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.811748 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.813281 (+ 1533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:28.813324 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:28.817527 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:28.822129 (+ 4602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:28.822138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:28.822173 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:28.822774 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:28.823481 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:28.823488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:28.823507 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:28.823813 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:28.824908 (+ 1095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:28.824927 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:28.825434 (+ 507us) client_negotiation.cc:769] Sending connection context
0810 14:35:28.825737 (+ 303us) client_negotiation.cc:240] Negotiation successful
0810 14:35:28.826295 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":152,"threads_started":1}
I20260810 14:35:28.829526 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23653
I20260810 14:35:28.829780 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:28.830278 21677 external_mini_cluster.cc:1428] Setting key 1e870891e8ea33403252cf761fd0ecc0
I20260810 14:35:28.831028 23794 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41847
I20260810 14:35:28.831341 23794 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:28.832510 23794 heartbeater.cc:511] Master 127.21.43.126:41847 requested a full tablet report, sending...
I20260810 14:35:28.835830 23572 ts_manager.cc:194] Registered new tserver with Master: bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703)
I20260810 14:35:28.838729 23572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:51899
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:28.852941 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.835829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.65:48427)
0810 14:35:28.836608 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:28.836638 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:28.836841 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:28.837421 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:28.837433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:28.837894 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:28.838164 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.838173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.840295 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.840301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.842191 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.842224 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.843544 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.843550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.843775 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.844812 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:28.844838 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:28.845519 (+ 681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:28.849747 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:28.849750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:28.849752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:28.850169 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:28.850799 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:28.850801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:28.850803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:28.850976 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:28.851980 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:28.851983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:28.852130 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 14:35:28.852354 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 14:35:28.852633 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":263,"threads_started":1}
I20260810 14:35:28.853080 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.836002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:48427 (local address 127.21.43.126:41847)
0810 14:35:28.836284 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:28.836291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:28.836784 (+ 493us) server_negotiation.cc:407] Connection header received
0810 14:35:28.837031 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:28.837050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:28.837157 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:28.837301 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:28.838423 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.840048 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.842498 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.843281 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.843529 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.845789 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:28.845807 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:28.845814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:28.845897 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:28.849493 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.850362 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.850381 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.850396 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.850597 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:28.851178 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:28.851192 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:28.851195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:28.851783 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:28.851985 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:28.852497 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 14:35:28.852763 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":108}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:28.884690 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41847
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:29.133301 23805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:29.133921 23805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:29.134106 23805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:29.141358 23805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:29.141475 23805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:29.141597 23805 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:35:29.141693 23805 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:29.141880 23805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:29.151628 23805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:41847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:29.154102 23805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:29.156243 23805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:29.174389 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.174419 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.174413 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:29.175041 23805 server_base.cc:1034] running on GCE node
I20260810 14:35:29.175990 23805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:29.177203 23805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:29.178781 23805 hybrid_clock.cc:648] HybridClock initialized: now 1786372529178751 us; error 107 us; skew 500 ppm
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:29.184357 23805 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:29.186084 23805 webserver.cc:460] Webserver started at http://127.21.43.66:37423/ using document root <none> and password file <none>
I20260810 14:35:29.187098 23805 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:29.187317 23805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:29.187624 23805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:29.192263 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "4f70725e4e7c4795987fd221b6bde4ab"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "f6f4e85c8bb8f2093997edf1124bd9ef"
server_key_iv: "85686f58e8ff00d0adcc96263973fa4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.193266 23805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "4f70725e4e7c4795987fd221b6bde4ab"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "f6f4e85c8bb8f2093997edf1124bd9ef"
server_key_iv: "85686f58e8ff00d0adcc96263973fa4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.201331 23805 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s
I20260810 14:35:29.207012 23822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.209782 23805 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 14:35:29.210178 23805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "4f70725e4e7c4795987fd221b6bde4ab"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "f6f4e85c8bb8f2093997edf1124bd9ef"
server_key_iv: "85686f58e8ff00d0adcc96263973fa4a"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.210458 23805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:29.241688 23805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:29.252473 23805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:29.252815 23805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:29.255908 23805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:29.256006 23805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.256121 23805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:29.256227 23805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:29.279198 23805 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:44507
I20260810 14:35:29.279373 23946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:44507 every 8 connection(s)
I20260810 14:35:29.280905 23805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:29.283025 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.257222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:45533 (local address 127.21.43.126:41847)
0810 14:35:29.257594 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.257601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.260021 (+ 2420us) server_negotiation.cc:407] Connection header received
0810 14:35:29.262281 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.262295 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.262423 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.262724 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.265166 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.266029 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.268814 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.269277 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.269572 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.275837 (+ 6265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.275841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.275844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.275885 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.278923 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.279984 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.279988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.279999 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.280126 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.280823 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.280836 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.280838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.281258 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.281476 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.282650 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.282840 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260810 14:35:29.284541 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23805
I20260810 14:35:29.284658 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.258790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.66:45533)
0810 14:35:29.259686 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.259794 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.261860 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.262975 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.263001 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.263838 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.264875 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.264909 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.266274 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.266281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.268616 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.268641 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.269719 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.269735 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.270267 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.271683 (+ 1416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.271746 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.275458 (+ 3712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.279268 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.279278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.279310 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.279783 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.280320 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.280335 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.280337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.280604 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.281460 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.281480 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.282160 (+ 680us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.282541 (+ 381us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.282973 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":279,"threads_started":1}
I20260810 14:35:29.284758 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:29.285288 21677 external_mini_cluster.cc:1428] Setting key dcdec276a192d82313bdc7db3861f3c5
I20260810 14:35:29.286926 23947 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41847
I20260810 14:35:29.287288 23947 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:29.288187 23947 heartbeater.cc:511] Master 127.21.43.126:41847 requested a full tablet report, sending...
I20260810 14:35:29.291962 23573 ts_manager.cc:194] Registered new tserver with Master: 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507)
I20260810 14:35:29.292982 23573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:45533
I20260810 14:35:29.307260 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.291479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:32985 (local address 127.21.43.126:41847)
0810 14:35:29.291788 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.291805 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.293972 (+ 2167us) server_negotiation.cc:407] Connection header received
0810 14:35:29.294304 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.294310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.294384 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.294540 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.295324 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.296134 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.297617 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.297978 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.298283 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.300586 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.300591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.300594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.300662 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.303502 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.304565 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.304570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.304572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.304730 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.305359 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.305378 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.305380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.305807 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.306065 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.306563 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.306815 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260810 14:35:29.307287 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.291241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.66:32985)
0810 14:35:29.293837 (+ 2596us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.293858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.294088 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.294592 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.294595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.295018 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.295196 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.295211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.296378 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.296394 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.297325 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.297337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.298297 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.298304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.298493 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.299566 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.299608 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.300282 (+ 674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.303847 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.303860 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.303863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.304333 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.304936 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.304954 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.304957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.305135 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.306013 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.306016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.306200 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.306448 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.306798 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2394,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:29.341503 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41847
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:29.584172 23958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:29.584861 23958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:29.584949 23958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:29.592609 23958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:29.592736 23958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:29.592830 23958 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:35:29.592857 23958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:29.592918 23958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:29.602772 23958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:41847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:29.605125 23958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:29.607381 23958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:29.628698 23963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.628672 23965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.628672 23967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:29.629596 23958 server_base.cc:1034] running on GCE node
I20260810 14:35:29.630548 23958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:29.632225 23958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:29.635636 23958 hybrid_clock.cc:648] HybridClock initialized: now 1786372529635582 us; error 62 us; skew 500 ppm
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:29.642023 23958 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:29.644030 23958 webserver.cc:460] Webserver started at http://127.21.43.67:44645/ using document root <none> and password file <none>
I20260810 14:35:29.645285 23958 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:29.645533 23958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:29.646149 23958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:29.650880 23958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "c87cca4849894dadb6bda662013bf08d"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "d281982f1b8df59ad837839d407f1b97"
server_key_iv: "9dad95f8287db8167ace3c6b74f0bec7"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.652129 23958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "c87cca4849894dadb6bda662013bf08d"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "d281982f1b8df59ad837839d407f1b97"
server_key_iv: "9dad95f8287db8167ace3c6b74f0bec7"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.660545 23958 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260810 14:35:29.666867 23975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.669893 23958 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260810 14:35:29.670120 23958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "c87cca4849894dadb6bda662013bf08d"
format_stamp: "Formatted at 2026-08-10 14:35:29 on dist-test-slave-chsk"
server_key: "d281982f1b8df59ad837839d407f1b97"
server_key_iv: "9dad95f8287db8167ace3c6b74f0bec7"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.670384 23958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:29.725446 23958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:29.733368 23958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:29.734012 23958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:29.738819 23958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:29.739050 23958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.739233 23958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:29.739324 23958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:29.767058 23958 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:39123
I20260810 14:35:29.767468 24106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:39123 every 8 connection(s)
I20260810 14:35:29.769352 23958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:29.771037 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 23958
I20260810 14:35:29.771421 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:29.772220 21677 external_mini_cluster.cc:1428] Setting key f8abb20531a7dfb0f21da9b76a5531bd
I20260810 14:35:29.774814 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.742327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45219 (local address 127.21.43.126:41847)
0810 14:35:29.742771 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.742779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.745237 (+ 2458us) server_negotiation.cc:407] Connection header received
0810 14:35:29.748168 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.748173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.748292 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.748441 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.753180 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.754861 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.759077 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.759518 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.759653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.765316 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.765328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.765332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.765397 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.768651 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.770038 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.770050 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.770051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.770248 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.771457 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.771462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.771464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.772099 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.772292 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.773965 (+ 1673us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.774312 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 14:35:29.776383 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.743523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.67:45219)
0810 14:35:29.744760 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.744915 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.747643 (+ 2728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.749328 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.749361 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.750725 (+ 1364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.752681 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.752756 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.755748 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.755767 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.758783 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.758808 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.759756 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.759770 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.760131 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.761274 (+ 1143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.761308 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.764789 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.768880 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.768900 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.768938 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.769786 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.770621 (+ 835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.770625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.770654 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.771068 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.772390 (+ 1322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.772395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.773008 (+ 613us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.773453 (+ 445us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.774196 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":310,"threads_started":1}
I20260810 14:35:29.779662 24107 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41847
I20260810 14:35:29.780032 24107 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:29.781090 24107 heartbeater.cc:511] Master 127.21.43.126:41847 requested a full tablet report, sending...
I20260810 14:35:29.784108 23573 ts_manager.cc:194] Registered new tserver with Master: c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:29.785908 23573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:45219
I20260810 14:35:29.791245 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:29.803879 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.785347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:59957 (local address 127.21.43.126:41847)
0810 14:35:29.785563 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.785568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.786005 (+ 437us) server_negotiation.cc:407] Connection header received
0810 14:35:29.786400 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.786415 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.786609 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.786812 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.789064 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.790552 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.792999 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.793386 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.793606 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.795829 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.795834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.795845 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.795892 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.799824 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.800908 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.800912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.800914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.801055 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.801829 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.801834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.801838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.802205 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.802414 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.803071 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.803429 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 14:35:29.803979 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.785124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.21.43.67:59957)
0810 14:35:29.785814 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.785838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.786088 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.786904 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.786907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.787656 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.788687 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.788718 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.791185 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.791189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.792661 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.792685 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.793611 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.793615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.793795 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.794709 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.794766 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.795520 (+ 754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.800164 (+ 4644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.800168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.800184 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.800700 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.801289 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.801302 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.801306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.801428 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.802495 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.802498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.802705 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.802927 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.803433 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":70,"thread_start_us":300,"threads_started":1}
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260810 14:35:29.844141 23794 heartbeater.cc:503] Master 127.21.43.126:41847 was elected leader, sending a full tablet report...
Aug 10 14:35:29 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:29.865355 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.842466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.21.43.126:41847)
0810 14:35:29.842873 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.842883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.843058 (+ 175us) server_negotiation.cc:407] Connection header received
0810 14:35:29.843510 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.843516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.843602 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.843851 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.845529 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.846748 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.848671 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.849188 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.849612 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.857256 (+ 7644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.857275 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.857278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.857362 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.860987 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.862311 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.862315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.862334 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.862497 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.863335 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.863339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.863348 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.863749 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.864068 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.864699 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.865016 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260810 14:35:29.871124 23573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:29.876987 23573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:29.913779 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.898473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:39123 (local address 127.0.0.1:44042)
0810 14:35:29.900987 (+ 2514us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.901134 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.902372 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.904140 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.904165 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.904288 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.904811 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.904854 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.906748 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.906757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.909656 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.909695 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.912263 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.912269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.912620 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.912701 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.913010 (+ 309us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.913335 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"mutex_wait_us":309,"spinlock_wait_cycles":27904,"thread_start_us":493,"threads_started":1}
I20260810 14:35:29.915042 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.900764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.0.0.1:41572)
0810 14:35:29.901785 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.901802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.902123 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.904141 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.904162 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.904294 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.904771 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.904780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.908967 (+ 4187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.908971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.911390 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.911405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.913812 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.913816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.914368 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.914422 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.914550 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.914739 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":138,"spinlock_wait_cycles":83968,"thread_start_us":466,"threads_started":1}
I20260810 14:35:29.915246 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.898621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.21.43.67:39123)
0810 14:35:29.900923 (+ 2302us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.900967 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.902388 (+ 1421us) server_negotiation.cc:407] Connection header received
0810 14:35:29.902756 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.902774 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.903363 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.903789 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.905205 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.906492 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.909939 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.911340 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.911619 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.912094 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.914560 (+ 2466us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.914807 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2045,"thread_start_us":355,"threads_started":1}
I20260810 14:35:29.915620 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.901642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.21.43.65:39703)
0810 14:35:29.902918 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.902948 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.903006 (+ 58us) server_negotiation.cc:407] Connection header received
0810 14:35:29.903314 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.903334 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.903796 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.904293 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.905048 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.908655 (+ 3607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.911907 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.913454 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.913820 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.914571 (+ 751us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.914693 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.914879 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":236,"threads_started":1}
I20260810 14:35:29.915926 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.898701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44507 (local address 127.0.0.1:56858)
0810 14:35:29.900987 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.901134 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.901460 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.903794 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.903813 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.903883 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.904482 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.904494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.907434 (+ 2940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.907439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.909963 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.909993 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.914569 (+ 4576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.914575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.914813 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.914850 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.915020 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.915310 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":295,"spinlock_wait_cycles":8064,"thread_start_us":257,"threads_started":1}
I20260810 14:35:29.917236 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.899247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.21.43.66:44507)
0810 14:35:29.902298 (+ 3051us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.902313 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.902387 (+ 74us) server_negotiation.cc:407] Connection header received
0810 14:35:29.902593 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.902622 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.903353 (+ 731us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.903789 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.904785 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.906951 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.910340 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.913754 (+ 3414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.914029 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.914469 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.916597 (+ 2128us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.916929 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2813,"thread_start_us":155,"threads_started":1}
I20260810 14:35:29.922950 23875 tablet_service.cc:1467] Processing CreateTablet for tablet f7942400b9004ac3b9436f08aa2218e0 (DEFAULT_TABLE table=test-table [id=9adab6d0d5f346da908bd2534798eac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.923098 24036 tablet_service.cc:1467] Processing CreateTablet for tablet f7942400b9004ac3b9436f08aa2218e0 (DEFAULT_TABLE table=test-table [id=9adab6d0d5f346da908bd2534798eac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.923094 23722 tablet_service.cc:1467] Processing CreateTablet for tablet f7942400b9004ac3b9436f08aa2218e0 (DEFAULT_TABLE table=test-table [id=9adab6d0d5f346da908bd2534798eac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.925751 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7942400b9004ac3b9436f08aa2218e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:29.925755 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7942400b9004ac3b9436f08aa2218e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:29.925944 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7942400b9004ac3b9436f08aa2218e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:29.940624 24132 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Bootstrap starting.
I20260810 14:35:29.944275 24133 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Bootstrap starting.
I20260810 14:35:29.945173 24134 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Bootstrap starting.
I20260810 14:35:29.947397 24132 tablet_bootstrap.cc:654] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:29.948573 24134 tablet_bootstrap.cc:654] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:29.948628 24133 tablet_bootstrap.cc:654] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:29.949615 24132 log.cc:824] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:29.949769 24134 log.cc:824] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:29.950181 24133 log.cc:824] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:29.955353 24132 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: No bootstrap required, opened a new log
I20260810 14:35:29.955369 24134 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: No bootstrap required, opened a new log
I20260810 14:35:29.955849 24134 ts_tablet_manager.cc:1397] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 14:35:29.956362 24132 ts_tablet_manager.cc:1397] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260810 14:35:29.958603 24133 tablet_bootstrap.cc:492] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: No bootstrap required, opened a new log
I20260810 14:35:29.958974 24133 ts_tablet_manager.cc:1397] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260810 14:35:29.962309 24134 raft_consensus.cc:348] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.962925 24134 raft_consensus.cc:374] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:29.963095 24134 raft_consensus.cc:729] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cca4849894dadb6bda662013bf08d, State: Initialized, Role: FOLLOWER
I20260810 14:35:29.963805 24134 consensus_queue.cc:260] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.965432 24107 heartbeater.cc:503] Master 127.21.43.126:41847 was elected leader, sending a full tablet report...
I20260810 14:35:29.966670 24134 ts_tablet_manager.cc:1428] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260810 14:35:29.967590 24132 raft_consensus.cc:348] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.968530 24132 raft_consensus.cc:374] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:29.968822 24132 raft_consensus.cc:729] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Initialized, Role: FOLLOWER
I20260810 14:35:29.970161 24132 consensus_queue.cc:260] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.969870 24133 raft_consensus.cc:348] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.970872 24133 raft_consensus.cc:374] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:29.971118 24133 raft_consensus.cc:729] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Initialized, Role: FOLLOWER
I20260810 14:35:29.972343 24133 consensus_queue.cc:260] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:29.973232 24132 ts_tablet_manager.cc:1428] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
I20260810 14:35:29.974005 23947 heartbeater.cc:503] Master 127.21.43.126:41847 was elected leader, sending a full tablet report...
I20260810 14:35:29.974920 24133 ts_tablet_manager.cc:1428] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Time spent starting tablet: real 0.016s user 0.014s sys 0.004s
W20260810 14:35:30.022855 24108 tablet.cc:2369] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:30.033674 23948 tablet.cc:2369] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:30.076766 23795 tablet.cc:2369] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:30.145722 24139 raft_consensus.cc:482] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.146014 24139 raft_consensus.cc:504] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:30.147400 24139 leader_election.cc:288] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123), bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703)
I20260810 14:35:30.151798 24140 raft_consensus.cc:482] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.152007 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.147680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:39123 (local address 127.21.43.66:54481)
0810 14:35:30.148253 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.148300 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.148423 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.148713 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.148715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.148730 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.148856 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.148863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.149757 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.149760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.150769 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.150778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.151518 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.151521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.151631 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.151656 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.151741 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.151812 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":204,"thread_start_us":150,"threads_started":1}
I20260810 14:35:30.152015 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.147792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:54481 (local address 127.21.43.67:39123)
0810 14:35:30.147962 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.147968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.148414 (+ 446us) server_negotiation.cc:407] Connection header received
0810 14:35:30.148560 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.148563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.148611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.148691 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.148991 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.149637 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.150895 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.151389 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.151524 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.151579 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.151782 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.151878 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:35:30.152123 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.147812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:51625 (local address 127.21.43.65:39703)
0810 14:35:30.147960 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.147966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.148326 (+ 360us) server_negotiation.cc:407] Connection header received
0810 14:35:30.148401 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.148404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.148455 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.148533 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.148854 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.149541 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.150880 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.151394 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.151574 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.151625 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.151899 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.151979 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:30.152513 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.147676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.21.43.66:51625)
0810 14:35:30.148193 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.148206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.148324 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.148559 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.148561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.148579 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.148727 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.148734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.149689 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.149695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.150749 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.150759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.151647 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.151650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.151764 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.151782 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.152280 (+ 498us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.152363 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":1,"thread_start_us":166,"threads_started":1}
I20260810 14:35:30.152643 24140 raft_consensus.cc:504] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:30.152812 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:30.153080 24046 raft_consensus.cc:2454] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 0.
I20260810 14:35:30.153152 23744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61a92bba04aab86519939b6cdf59a" is_pre_election: true
I20260810 14:35:30.153386 23744 raft_consensus.cc:2454] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 0.
I20260810 14:35:30.153620 23823 leader_election.cc:302] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, c87cca4849894dadb6bda662013bf08d; no voters:
I20260810 14:35:30.153916 24139 raft_consensus.cc:2781] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:30.154019 24139 raft_consensus.cc:482] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:30.154079 24139 raft_consensus.cc:3037] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.154623 24140 leader_election.cc:288] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123), 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507)
I20260810 14:35:30.155651 24139 raft_consensus.cc:504] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:30.156066 24139 leader_election.cc:288] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 election: Requested vote from peers c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123), bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703)
I20260810 14:35:30.156466 23744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61a92bba04aab86519939b6cdf59a"
I20260810 14:35:30.156601 23744 raft_consensus.cc:3037] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.157888 23744 raft_consensus.cc:2454] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 1.
I20260810 14:35:30.158180 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d"
I20260810 14:35:30.158313 23827 leader_election.cc:302] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a; no voters:
I20260810 14:35:30.158331 24046 raft_consensus.cc:3037] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.158519 24139 raft_consensus.cc:2781] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:30.158753 24139 raft_consensus.cc:686] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 LEADER]: Becoming Leader. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Running, Role: LEADER
I20260810 14:35:30.159179 24139 consensus_queue.cc:237] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } }
I20260810 14:35:30.159462 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.154680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:50933 (local address 127.21.43.66:44507)
0810 14:35:30.154803 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.154814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.155492 (+ 678us) server_negotiation.cc:407] Connection header received
0810 14:35:30.155682 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.155685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.155737 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.155811 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.156393 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.156987 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.158106 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.158716 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.158807 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.158869 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.159253 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.159335 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:30.159906 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.154281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44507 (local address 127.21.43.65:50933)
0810 14:35:30.155405 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.155421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.155563 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.155910 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.155913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.155936 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.156251 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.156261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.157100 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.157106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.157932 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.157942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.158969 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.158972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.159087 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.159108 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.159667 (+ 559us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.159753 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":25,"thread_start_us":134,"threads_started":1}
I20260810 14:35:30.160193 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.155129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:55485 (local address 127.21.43.67:39123)
0810 14:35:30.155290 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.155294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.155308 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:35:30.155351 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.155354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.155402 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.155494 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.155852 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.156572 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.158121 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.158980 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.159084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.159173 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.159805 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.159875 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 14:35:30.160288 24046 raft_consensus.cc:2454] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 1.
I20260810 14:35:30.160521 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab" is_pre_election: true
I20260810 14:35:30.160674 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.154270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:39123 (local address 127.21.43.65:55485)
0810 14:35:30.154840 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.154854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.154946 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.155511 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.155514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.155532 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.155681 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.155688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.156710 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.156714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.157997 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.158007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.159243 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.159247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.159424 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.159448 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.159708 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.159760 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":124,"threads_started":1}
I20260810 14:35:30.161128 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7942400b9004ac3b9436f08aa2218e0" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:30.161330 24046 raft_consensus.cc:2379] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bce61a92bba04aab86519939b6cdf59a in current term 1: Already voted for candidate 4f70725e4e7c4795987fd221b6bde4ab in this term.
I20260810 14:35:30.161754 23672 leader_election.cc:302] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bce61a92bba04aab86519939b6cdf59a; no voters: 4f70725e4e7c4795987fd221b6bde4ab, c87cca4849894dadb6bda662013bf08d
I20260810 14:35:30.162058 24140 raft_consensus.cc:2726] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 14:35:30.163892 23572 catalog_manager.cc:5495] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66). New cstate: current_term: 1 leader_uuid: "4f70725e4e7c4795987fd221b6bde4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:35:30 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:30.241739 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.233159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41847 (local address 127.0.0.1:57444)
0810 14:35:30.233475 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.233489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.233598 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.233841 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.233844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.234237 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:30.234352 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.234359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.235064 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.235066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.235627 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.235635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.236061 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.236063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.236134 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.236668 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:30.236680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:30.238257 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:30.240157 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:30.240166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:30.240172 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:30.240418 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:30.240733 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:30.240735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:30.240737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:30.240815 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:30.241248 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:30.241253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:30.241379 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.241471 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.241591 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260810 14:35:30.241782 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.233312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.21.43.126:41847)
0810 14:35:30.233501 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.233505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.233567 (+ 62us) server_negotiation.cc:407] Connection header received
0810 14:35:30.233683 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.233685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.233727 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.233798 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:30.234464 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.234971 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.235735 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.235940 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.236037 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.238396 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:30.238400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:30.238402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:30.238424 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:30.240049 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.240541 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.240544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.240546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.240634 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.240901 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.240904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.240905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.241132 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:30.241226 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.241554 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.241664 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 14:35:30.246022 23572 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57444:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:35:30.246779 23572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:30.252269 24036 tablet_service.cc:1467] Processing CreateTablet for tablet b329d2d1d4aa481eb5018a250b4852e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be4d9ea607a413e9d68454262c1de28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:30.252137 23722 tablet_service.cc:1467] Processing CreateTablet for tablet b329d2d1d4aa481eb5018a250b4852e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be4d9ea607a413e9d68454262c1de28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:30.252148 23875 tablet_service.cc:1467] Processing CreateTablet for tablet b329d2d1d4aa481eb5018a250b4852e1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be4d9ea607a413e9d68454262c1de28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:30.252902 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b329d2d1d4aa481eb5018a250b4852e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.252902 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b329d2d1d4aa481eb5018a250b4852e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.252903 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b329d2d1d4aa481eb5018a250b4852e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.255604 24134 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: Bootstrap starting.
I20260810 14:35:30.256124 24133 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: Bootstrap starting.
I20260810 14:35:30.256717 24134 tablet_bootstrap.cc:654] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.256759 24132 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: Bootstrap starting.
I20260810 14:35:30.257407 24133 tablet_bootstrap.cc:654] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.258201 24132 tablet_bootstrap.cc:654] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.258263 24134 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: No bootstrap required, opened a new log
I20260810 14:35:30.258343 24134 ts_tablet_manager.cc:1397] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:30.258870 24134 raft_consensus.cc:348] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.258991 24134 raft_consensus.cc:374] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.259029 24134 raft_consensus.cc:729] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cca4849894dadb6bda662013bf08d, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.259232 24134 consensus_queue.cc:260] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.259573 24132 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: No bootstrap required, opened a new log
I20260810 14:35:30.259554 24138 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.259636 24133 tablet_bootstrap.cc:492] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: No bootstrap required, opened a new log
I20260810 14:35:30.259651 24132 ts_tablet_manager.cc:1397] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 14:35:30.259712 24138 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.259718 24133 ts_tablet_manager.cc:1397] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:30.260159 24132 raft_consensus.cc:348] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.260247 24134 ts_tablet_manager.cc:1428] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 14:35:30.260202 24133 raft_consensus.cc:348] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.260295 24132 raft_consensus.cc:374] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.260319 24133 raft_consensus.cc:374] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.260340 24132 raft_consensus.cc:729] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.260352 24133 raft_consensus.cc:729] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.260473 24132 consensus_queue.cc:260] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.260519 24133 consensus_queue.cc:260] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.260776 24132 ts_tablet_manager.cc:1428] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:35:30.260773 24140 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.260890 24140 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.261452 24133 ts_tablet_manager.cc:1428] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:30.260773 24139 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.262194 24139 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.382157 24138 raft_consensus.cc:482] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.382354 24138 raft_consensus.cc:504] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.383723 24138 leader_election.cc:288] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703), 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507)
I20260810 14:35:30.384124 24140 raft_consensus.cc:482] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.384243 24140 raft_consensus.cc:504] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.384681 24140 leader_election.cc:288] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:30.385212 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab" is_pre_election: true
I20260810 14:35:30.385308 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:30.385433 24046 raft_consensus.cc:2454] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:30.385434 23896 raft_consensus.cc:2454] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:30.385804 23672 leader_election.cc:302] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bce61a92bba04aab86519939b6cdf59a, c87cca4849894dadb6bda662013bf08d; no voters:
I20260810 14:35:30.386067 24140 raft_consensus.cc:2781] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:30.386157 24140 raft_consensus.cc:482] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:30.386220 24140 raft_consensus.cc:3037] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.387705 24140 raft_consensus.cc:504] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.388170 24140 leader_election.cc:288] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Requested vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:30.388566 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab"
I20260810 14:35:30.388736 23896 raft_consensus.cc:3037] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.389214 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.384098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:39837 (local address 127.21.43.65:39703)
0810 14:35:30.384269 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.384274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.384417 (+ 143us) server_negotiation.cc:407] Connection header received
0810 14:35:30.384632 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.384636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.384693 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.384752 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.385235 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.386017 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.387856 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.388557 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.388697 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.388759 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.388992 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.389081 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:30.389233 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.383928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.21.43.67:39837)
0810 14:35:30.384321 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.384337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.384507 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.384918 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.384923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.384948 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.385092 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.385099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.386169 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.386172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.387668 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.387676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.388703 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.388706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.388835 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.388867 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.388954 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.389048 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":94,"thread_start_us":123,"threads_started":1}
I20260810 14:35:30.389575 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d"
I20260810 14:35:30.389621 23744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "c87cca4849894dadb6bda662013bf08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61a92bba04aab86519939b6cdf59a" is_pre_election: true
I20260810 14:35:30.389712 24046 raft_consensus.cc:3037] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.389807 23744 raft_consensus.cc:2379] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c87cca4849894dadb6bda662013bf08d in current term 1: Already voted for candidate bce61a92bba04aab86519939b6cdf59a in this term.
I20260810 14:35:30.389959 23896 raft_consensus.cc:2454] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:30.390658 23674 leader_election.cc:302] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a; no voters:
I20260810 14:35:30.390874 24140 raft_consensus.cc:2781] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:30.391048 24046 raft_consensus.cc:2454] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:30.391206 24140 raft_consensus.cc:686] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [term 1 LEADER]: Becoming Leader. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Running, Role: LEADER
I20260810 14:35:30.391639 24140 consensus_queue.cc:237] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.391891 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.384545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:58537 (local address 127.21.43.66:44507)
0810 14:35:30.384695 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.384700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.384718 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:35:30.384917 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.384923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.384983 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.385105 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.385464 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.386170 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.388539 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.389222 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.389361 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.389432 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.391685 (+ 2253us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.391772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:35:30.391896 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.384094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44507 (local address 127.21.43.67:58537)
0810 14:35:30.384612 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.384629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.384740 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.385217 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.385220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.385238 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.385331 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.385336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.386285 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.386288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.387664 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.387676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.391327 (+ 3651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.391331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.391443 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.391491 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.391642 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.391736 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":108,"thread_start_us":104,"threads_started":1}
I20260810 14:35:30.392386 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b329d2d1d4aa481eb5018a250b4852e1" candidate_uuid: "c87cca4849894dadb6bda662013bf08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab" is_pre_election: true
I20260810 14:35:30.392563 23896 raft_consensus.cc:2379] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c87cca4849894dadb6bda662013bf08d in current term 1: Already voted for candidate bce61a92bba04aab86519939b6cdf59a in this term.
I20260810 14:35:30.392871 23978 leader_election.cc:302] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c87cca4849894dadb6bda662013bf08d; no voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a
I20260810 14:35:30.393132 24138 raft_consensus.cc:2726] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 14:35:30.394479 24159 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: TxnStatusTablet state changed. Reason: New leader bce61a92bba04aab86519939b6cdf59a. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.394614 24159 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:30.394920 24161 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:35:30.395187 24161 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:35:30.396009 23572 catalog_manager.cc:5495] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a reported cstate change: term changed from 0 to 1, leader changed from <none> to bce61a92bba04aab86519939b6cdf59a (127.21.43.65). New cstate: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:30.417719 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.413709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.21.43.65:39703)
0810 14:35:30.413830 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.413837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.413879 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:35:30.414122 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.414124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.414190 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.414299 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.414557 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.415093 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.416113 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.416354 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.416464 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.416820 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.417298 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.417429 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.417502 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.417561 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:30.417719 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.413599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.0.0.1:41578)
0810 14:35:30.413807 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.413824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.413959 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.414313 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.414317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.414324 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:30.414447 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.414453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.415260 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.415264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.415988 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.415999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.416481 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.416484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.416577 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.416649 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.417402 (+ 753us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.417420 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.417492 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.417584 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 14:35:30.423938 24046 raft_consensus.cc:1261] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.423990 23896 raft_consensus.cc:1261] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.424571 24140 consensus_queue.cc:1035] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.424778 24159 consensus_queue.cc:1035] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.428061 24139 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: TxnStatusTablet state changed. Reason: New leader bce61a92bba04aab86519939b6cdf59a. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.428193 24139 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.429375 24138 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: TxnStatusTablet state changed. Reason: New leader bce61a92bba04aab86519939b6cdf59a. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.429515 24138 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.437738 24140 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.437896 24140 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:30.437976 24159 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.438094 24159 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P bce61a92bba04aab86519939b6cdf59a: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:30.438421 24139 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.438536 24139 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P 4f70725e4e7c4795987fd221b6bde4ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.439000 24138 tablet_replica.cc:439] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } } }
I20260810 14:35:30.439121 24138 tablet_replica.cc:442] T b329d2d1d4aa481eb5018a250b4852e1 P c87cca4849894dadb6bda662013bf08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:30.440004 24166 mvcc.cc:205] Tried to move back new op lower bound from 7316981884608499712 to 7316981884488409088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981884608499712}]
I20260810 14:35:30.440922 24165 mvcc.cc:205] Tried to move back new op lower bound from 7316981884608499712 to 7316981884488409088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981884608499712}]
I20260810 14:35:30.441387 24164 mvcc.cc:205] Tried to move back new op lower bound from 7316981884608499712 to 7316981884488409088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981884608499712}]
I20260810 14:35:30.451910 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.447443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.21.43.66:44507)
0810 14:35:30.447570 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.447575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.447763 (+ 188us) server_negotiation.cc:407] Connection header received
0810 14:35:30.447916 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.447919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.447988 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.448111 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.448360 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.449164 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.450033 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.450385 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.450491 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.450751 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.451344 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.451460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.451688 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.451756 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:30.463863 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.459998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:42551 (local address 127.21.43.65:39703)
0810 14:35:30.460160 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.460166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.460356 (+ 190us) server_negotiation.cc:407] Connection header received
0810 14:35:30.460464 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.460467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.460515 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.460588 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.460867 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.461438 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.462707 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.462942 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.463032 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.463453 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.463527 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.463591 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.463707 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.463751 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:35:30.463959 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.459895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.21.43.66:42551)
0810 14:35:30.460254 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.460269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.460378 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.460619 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.460622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.460630 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:30.460760 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.460766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.461544 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.461547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.462561 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.462573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.463060 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.463064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.463212 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.463267 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.463620 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.463626 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.463701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.463784 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
I20260810 14:35:30.471324 23744 raft_consensus.cc:1261] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Refusing update from remote peer 4f70725e4e7c4795987fd221b6bde4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.471521 24046 raft_consensus.cc:1261] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Refusing update from remote peer 4f70725e4e7c4795987fd221b6bde4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.471885 24146 consensus_queue.cc:1035] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.472118 24139 consensus_queue.cc:1035] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.497231 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.493164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:48953 (local address 127.21.43.66:44507)
0810 14:35:30.493267 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.493272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.493286 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:35:30.493376 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.493379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.493427 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.493503 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.493803 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.494582 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.495867 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.496205 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.496281 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.496726 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.496816 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.496858 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.497013 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.497077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:35:30.497704 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.492792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44507 (local address 127.21.43.65:48953)
0810 14:35:30.493152 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.493170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.493272 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.493529 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.493532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.493541 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:30.493687 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.493693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.494707 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.494711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.495717 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.495729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.496397 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.496401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.496498 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.496556 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.496919 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.496925 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.497462 (+ 537us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.497552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260810 14:35:30.528321 23573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:30.528551 23573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:30.531605 23573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Sending DeleteTablet for 3 replicas of tablet f7942400b9004ac3b9436f08aa2218e0
I20260810 14:35:30.532500 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet f7942400b9004ac3b9436f08aa2218e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:44042
I20260810 14:35:30.532606 23875 tablet_service.cc:1514] Processing DeleteTablet for tablet f7942400b9004ac3b9436f08aa2218e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:56858
I20260810 14:35:30.533418 23722 tablet_service.cc:1514] Processing DeleteTablet for tablet f7942400b9004ac3b9436f08aa2218e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41572
I20260810 14:35:30.533510 24187 tablet_replica.cc:330] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: stopping tablet replica
I20260810 14:35:30.533874 24187 raft_consensus.cc:2229] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:30.534140 24187 raft_consensus.cc:2258] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.534664 23573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:30.535173 23573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:30.535972 24188 tablet_replica.cc:330] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: stopping tablet replica
I20260810 14:35:30.536301 24188 raft_consensus.cc:2229] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:30.536718 24188 raft_consensus.cc:2258] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.539403 24188 ts_tablet_manager.cc:1905] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:30.541129 24187 ts_tablet_manager.cc:1905] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:30.542199 24190 tablet_replica.cc:330] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: stopping tablet replica
I20260810 14:35:30.542636 24190 raft_consensus.cc:2229] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:30.542673 23875 tablet_service.cc:1467] Processing CreateTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b (DEFAULT_TABLE table=test-table [id=d44b7321bb0b4c5e99469f95cbae4523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:30.542969 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f722f02ca9342d0ad4aebe2aa544f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.543509 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b (DEFAULT_TABLE table=test-table [id=d44b7321bb0b4c5e99469f95cbae4523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:30.543128 23722 tablet_service.cc:1467] Processing CreateTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b (DEFAULT_TABLE table=test-table [id=d44b7321bb0b4c5e99469f95cbae4523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:30.543816 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f722f02ca9342d0ad4aebe2aa544f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.543918 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f722f02ca9342d0ad4aebe2aa544f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.546267 24134 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Bootstrap starting.
I20260810 14:35:30.546449 24132 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Bootstrap starting.
I20260810 14:35:30.547319 24134 tablet_bootstrap.cc:654] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.547480 24132 tablet_bootstrap.cc:654] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.548445 24133 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Bootstrap starting.
I20260810 14:35:30.548635 24134 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: No bootstrap required, opened a new log
I20260810 14:35:30.548722 24134 ts_tablet_manager.cc:1397] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:35:30.548942 24188 ts_tablet_manager.cc:1918] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:30.549063 24188 log.cc:1196] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/f7942400b9004ac3b9436f08aa2218e0
I20260810 14:35:30.548965 24190 raft_consensus.cc:2258] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.549230 24134 raft_consensus.cc:348] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.549367 24134 raft_consensus.cc:374] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.549407 24188 ts_tablet_manager.cc:1939] T f7942400b9004ac3b9436f08aa2218e0 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting consensus metadata
I20260810 14:35:30.549414 24134 raft_consensus.cc:729] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cca4849894dadb6bda662013bf08d, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.549458 24133 tablet_bootstrap.cc:654] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.549557 24134 consensus_queue.cc:260] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.550652 23558 catalog_manager.cc:4841] TS 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507): tablet f7942400b9004ac3b9436f08aa2218e0 (table test-table [id=9adab6d0d5f346da908bd2534798eac4]) successfully deleted
I20260810 14:35:30.550740 24134 ts_tablet_manager.cc:1428] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:35:30.551923 24187 ts_tablet_manager.cc:1918] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:30.552034 24187 log.cc:1196] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/f7942400b9004ac3b9436f08aa2218e0
I20260810 14:35:30.552402 24187 ts_tablet_manager.cc:1939] T f7942400b9004ac3b9436f08aa2218e0 P c87cca4849894dadb6bda662013bf08d: Deleting consensus metadata
I20260810 14:35:30.552764 24190 ts_tablet_manager.cc:1905] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:30.553666 23556 catalog_manager.cc:4841] TS c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123): tablet f7942400b9004ac3b9436f08aa2218e0 (table test-table [id=9adab6d0d5f346da908bd2534798eac4]) successfully deleted
I20260810 14:35:30.554183 24133 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: No bootstrap required, opened a new log
I20260810 14:35:30.554292 24133 ts_tablet_manager.cc:1397] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 14:35:30.554832 24133 raft_consensus.cc:348] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.554971 24133 raft_consensus.cc:374] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.555011 24133 raft_consensus.cc:729] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.555177 24133 consensus_queue.cc:260] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.555487 24133 ts_tablet_manager.cc:1428] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:30.556802 24190 ts_tablet_manager.cc:1918] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:30.556910 24190 log.cc:1196] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/f7942400b9004ac3b9436f08aa2218e0
I20260810 14:35:30.557736 24132 tablet_bootstrap.cc:492] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: No bootstrap required, opened a new log
I20260810 14:35:30.557852 24132 ts_tablet_manager.cc:1397] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260810 14:35:30.558360 24132 raft_consensus.cc:348] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.558487 24132 raft_consensus.cc:374] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.558519 24132 raft_consensus.cc:729] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.558646 24132 consensus_queue.cc:260] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.558914 24132 ts_tablet_manager.cc:1428] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:30.559218 24190 ts_tablet_manager.cc:1939] T f7942400b9004ac3b9436f08aa2218e0 P bce61a92bba04aab86519939b6cdf59a: Deleting consensus metadata
I20260810 14:35:30.561509 23560 catalog_manager.cc:4841] TS bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703): tablet f7942400b9004ac3b9436f08aa2218e0 (table test-table [id=9adab6d0d5f346da908bd2534798eac4]) successfully deleted
I20260810 14:35:30.725818 24170 raft_consensus.cc:482] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.725991 24170 raft_consensus.cc:504] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.726543 24170 leader_election.cc:288] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:30.727159 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f722f02ca9342d0ad4aebe2aa544f1b" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:30.727159 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f722f02ca9342d0ad4aebe2aa544f1b" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab" is_pre_election: true
I20260810 14:35:30.727334 23896 raft_consensus.cc:2454] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:30.727334 24046 raft_consensus.cc:2454] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:30.727669 23674 leader_election.cc:302] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a; no voters:
I20260810 14:35:30.727900 24170 raft_consensus.cc:2781] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:30.727972 24170 raft_consensus.cc:482] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:30.728003 24170 raft_consensus.cc:3037] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.729120 24170 raft_consensus.cc:504] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.729534 24170 leader_election.cc:288] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Requested vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:30.729913 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f722f02ca9342d0ad4aebe2aa544f1b" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab"
I20260810 14:35:30.730057 23896 raft_consensus.cc:3037] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.730048 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f722f02ca9342d0ad4aebe2aa544f1b" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d"
I20260810 14:35:30.730219 24046 raft_consensus.cc:3037] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.731096 23896 raft_consensus.cc:2454] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:30.731281 24046 raft_consensus.cc:2454] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:30.731526 23674 leader_election.cc:302] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a; no voters:
I20260810 14:35:30.731714 24170 raft_consensus.cc:2781] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:30.731833 24170 raft_consensus.cc:686] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 1 LEADER]: Becoming Leader. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Running, Role: LEADER
I20260810 14:35:30.731987 24170 consensus_queue.cc:237] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:30.733899 23573 catalog_manager.cc:5495] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a reported cstate change: term changed from 0 to 1, leader changed from <none> to bce61a92bba04aab86519939b6cdf59a (127.21.43.65). New cstate: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:30.755448 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.752655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.21.43.65:39703)
0810 14:35:30.752791 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.752800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.752815 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:35:30.752859 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.752861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.752903 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.752973 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.753164 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.753655 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.754478 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.754695 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.754786 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.755015 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.755083 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.755201 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.755294 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.755338 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:35:30.764052 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.760989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.21.43.65:39913)
0810 14:35:30.761130 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.761164 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.761246 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.761460 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.761463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.761469 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:30.761581 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.761587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.762319 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.762322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.762879 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.762887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.763338 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.763341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.763489 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.763518 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.763790 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.763797 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.763856 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.763933 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":63}
I20260810 14:35:30.764053 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.761109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:39913 (local address 127.21.43.65:39703)
0810 14:35:30.761223 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.761227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.761239 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:35:30.761338 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.761341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.761380 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.761455 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.761714 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.762207 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.762998 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.763226 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.763329 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.763637 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.763703 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.763777 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.763864 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.763905 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":48}
I20260810 14:35:30.770115 24046 raft_consensus.cc:1261] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.770263 23896 raft_consensus.cc:1261] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.770566 24140 consensus_queue.cc:1035] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.770764 24159 consensus_queue.cc:1035] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:35:30.786448 24164 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:35:32.347779 23572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:32.347932 23572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:32.350345 23572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Sending DeleteTablet for 3 replicas of tablet 0f722f02ca9342d0ad4aebe2aa544f1b
I20260810 14:35:32.350925 23722 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41572
I20260810 14:35:32.350971 23875 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:56858
I20260810 14:35:32.351305 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f722f02ca9342d0ad4aebe2aa544f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:44042
I20260810 14:35:32.351583 24225 tablet_replica.cc:330] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: stopping tablet replica
I20260810 14:35:32.352078 24225 raft_consensus.cc:2229] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:32.352216 24225 raft_consensus.cc:2258] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:32.352873 24225 ts_tablet_manager.cc:1905] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.353837 24226 tablet_replica.cc:330] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: stopping tablet replica
I20260810 14:35:32.353995 24226 raft_consensus.cc:2229] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:32.354133 24226 raft_consensus.cc:2258] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:32.354830 24224 tablet_replica.cc:330] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: stopping tablet replica
I20260810 14:35:32.354864 24226 ts_tablet_manager.cc:1905] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.354995 24224 raft_consensus.cc:2229] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:32.355343 24224 raft_consensus.cc:2258] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:32.355953 24224 ts_tablet_manager.cc:1905] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.356314 24225 ts_tablet_manager.cc:1918] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:35:32.356406 24225 log.cc:1196] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/0f722f02ca9342d0ad4aebe2aa544f1b
I20260810 14:35:32.356730 24225 ts_tablet_manager.cc:1939] T 0f722f02ca9342d0ad4aebe2aa544f1b P 4f70725e4e7c4795987fd221b6bde4ab: Deleting consensus metadata
I20260810 14:35:32.357487 23558 catalog_manager.cc:4841] TS 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507): tablet 0f722f02ca9342d0ad4aebe2aa544f1b (table test-table [id=d44b7321bb0b4c5e99469f95cbae4523]) successfully deleted
I20260810 14:35:32.357647 24226 ts_tablet_manager.cc:1918] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:35:32.357736 24226 log.cc:1196] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/0f722f02ca9342d0ad4aebe2aa544f1b
I20260810 14:35:32.358021 24226 ts_tablet_manager.cc:1939] T 0f722f02ca9342d0ad4aebe2aa544f1b P c87cca4849894dadb6bda662013bf08d: Deleting consensus metadata
I20260810 14:35:32.358711 23556 catalog_manager.cc:4841] TS c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123): tablet 0f722f02ca9342d0ad4aebe2aa544f1b (table test-table [id=d44b7321bb0b4c5e99469f95cbae4523]) successfully deleted
I20260810 14:35:32.358876 24224 ts_tablet_manager.cc:1918] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:35:32.358963 24224 log.cc:1196] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/0f722f02ca9342d0ad4aebe2aa544f1b
I20260810 14:35:32.359284 24224 ts_tablet_manager.cc:1939] T 0f722f02ca9342d0ad4aebe2aa544f1b P bce61a92bba04aab86519939b6cdf59a: Deleting consensus metadata
I20260810 14:35:32.360026 23560 catalog_manager.cc:4841] TS bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703): tablet 0f722f02ca9342d0ad4aebe2aa544f1b (table test-table [id=d44b7321bb0b4c5e99469f95cbae4523]) successfully deleted
Aug 10 14:35:32 dist-test-slave-chsk krb5kdc[23526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:35:32 dist-test-slave-chsk krb5kdc[23526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372532, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:32.383427 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:32.374486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.21.43.126:41847)
0810 14:35:32.374808 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:32.374819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:32.374835 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:32.375011 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:32.375014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:32.375060 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:32.375179 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:32.375877 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.376430 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.377277 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.377510 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.377631 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:32.380290 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:32.380294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:32.380297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:32.380344 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:32.381651 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:32.382088 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:32.382092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:32.382095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:32.382187 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:32.382476 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:32.382479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:32.382481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:32.382676 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:32.382821 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:32.383065 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 14:35:32.383241 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260810 14:35:32.385918 23572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:32.386370 23572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:32.391409 23722 tablet_service.cc:1467] Processing CreateTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff (DEFAULT_TABLE table=test-table [id=cfe22355a7be47e7a940c2194236e343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.391466 23875 tablet_service.cc:1467] Processing CreateTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff (DEFAULT_TABLE table=test-table [id=cfe22355a7be47e7a940c2194236e343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.391774 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb75dfbb7c294a72bf8989083e8f32ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.391805 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb75dfbb7c294a72bf8989083e8f32ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.393903 24236 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Bootstrap starting.
I20260810 14:35:32.394145 24036 tablet_service.cc:1467] Processing CreateTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff (DEFAULT_TABLE table=test-table [id=cfe22355a7be47e7a940c2194236e343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.394435 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb75dfbb7c294a72bf8989083e8f32ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.394850 24236 tablet_bootstrap.cc:654] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.395479 24237 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Bootstrap starting.
I20260810 14:35:32.396251 24237 tablet_bootstrap.cc:654] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.396752 24239 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Bootstrap starting.
I20260810 14:35:32.397610 24237 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: No bootstrap required, opened a new log
I20260810 14:35:32.397627 24236 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: No bootstrap required, opened a new log
I20260810 14:35:32.397709 24237 ts_tablet_manager.cc:1397] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:32.397713 24236 ts_tablet_manager.cc:1397] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:32.397815 24239 tablet_bootstrap.cc:654] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.398139 24237 raft_consensus.cc:348] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.398168 24236 raft_consensus.cc:348] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.398263 24237 raft_consensus.cc:374] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.398277 24236 raft_consensus.cc:374] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.398306 24237 raft_consensus.cc:729] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.398316 24236 raft_consensus.cc:729] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.398432 24237 consensus_queue.cc:260] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.398491 24236 consensus_queue.cc:260] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.398857 24237 ts_tablet_manager.cc:1428] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:32.398888 24236 ts_tablet_manager.cc:1428] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:32.399080 24239 tablet_bootstrap.cc:492] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: No bootstrap required, opened a new log
I20260810 14:35:32.399192 24239 ts_tablet_manager.cc:1397] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:32.399650 24239 raft_consensus.cc:348] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.399772 24239 raft_consensus.cc:374] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.399816 24239 raft_consensus.cc:729] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cca4849894dadb6bda662013bf08d, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.399941 24239 consensus_queue.cc:260] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.400354 24239 ts_tablet_manager.cc:1428] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:32.549293 24210 raft_consensus.cc:482] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:32.549513 24210 raft_consensus.cc:504] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.550113 24210 leader_election.cc:288] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:32.550819 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb75dfbb7c294a72bf8989083e8f32ff" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:32.551082 24046 raft_consensus.cc:2454] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:32.550958 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb75dfbb7c294a72bf8989083e8f32ff" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab" is_pre_election: true
I20260810 14:35:32.551285 23896 raft_consensus.cc:2454] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 0.
I20260810 14:35:32.551836 23672 leader_election.cc:302] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bce61a92bba04aab86519939b6cdf59a, c87cca4849894dadb6bda662013bf08d; no voters:
I20260810 14:35:32.552130 24210 raft_consensus.cc:2781] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:32.552198 24210 raft_consensus.cc:482] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:32.552227 24210 raft_consensus.cc:3037] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.553304 24210 raft_consensus.cc:504] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.553747 24210 leader_election.cc:288] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Requested vote from peers 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:32.554265 23896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb75dfbb7c294a72bf8989083e8f32ff" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70725e4e7c4795987fd221b6bde4ab"
I20260810 14:35:32.554347 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb75dfbb7c294a72bf8989083e8f32ff" candidate_uuid: "bce61a92bba04aab86519939b6cdf59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d"
I20260810 14:35:32.554425 23896 raft_consensus.cc:3037] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.554527 24046 raft_consensus.cc:3037] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.555764 24046 raft_consensus.cc:2454] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:32.555764 23896 raft_consensus.cc:2454] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce61a92bba04aab86519939b6cdf59a in term 1.
I20260810 14:35:32.556135 23672 leader_election.cc:302] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bce61a92bba04aab86519939b6cdf59a, c87cca4849894dadb6bda662013bf08d; no voters:
I20260810 14:35:32.556339 24210 raft_consensus.cc:2781] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:32.556456 24210 raft_consensus.cc:686] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 1 LEADER]: Becoming Leader. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Running, Role: LEADER
I20260810 14:35:32.556622 24210 consensus_queue.cc:237] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.558739 23573 catalog_manager.cc:5495] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a reported cstate change: term changed from 0 to 1, leader changed from <none> to bce61a92bba04aab86519939b6cdf59a (127.21.43.65). New cstate: current_term: 1 leader_uuid: "bce61a92bba04aab86519939b6cdf59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:32.595613 24248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:32.592444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.21.43.65:39703)
0810 14:35:32.592743 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:32.592750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:32.592762 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:35:32.592804 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:32.592806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:32.592849 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:32.592923 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:32.593133 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.593646 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.594490 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.594722 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.594806 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:32.595043 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:32.595120 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:32.595345 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:32.595428 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:35:32.595479 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":139,"threads_started":1}
I20260810 14:35:32.602759 23896 raft_consensus.cc:1261] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:32.602814 24046 raft_consensus.cc:1261] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Refusing update from remote peer bce61a92bba04aab86519939b6cdf59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:32.603245 24210 consensus_queue.cc:1035] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:32.603472 24205 consensus_queue.cc:1035] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:32.639048 23573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:32.639286 23573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:32.641542 23573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Sending DeleteTablet for 3 replicas of tablet eb75dfbb7c294a72bf8989083e8f32ff
I20260810 14:35:32.642239 23722 tablet_service.cc:1514] Processing DeleteTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41572
I20260810 14:35:32.642386 23875 tablet_service.cc:1514] Processing DeleteTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:56858
I20260810 14:35:32.642484 24224 tablet_replica.cc:330] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: stopping tablet replica
I20260810 14:35:32.642578 24225 tablet_replica.cc:330] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: stopping tablet replica
I20260810 14:35:32.642623 24224 raft_consensus.cc:2229] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:32.642669 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet eb75dfbb7c294a72bf8989083e8f32ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:44042
I20260810 14:35:32.642711 24225 raft_consensus.cc:2229] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:32.642843 24225 raft_consensus.cc:2258] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:32.642858 24224 raft_consensus.cc:2258] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:32.642961 24226 tablet_replica.cc:330] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: stopping tablet replica
I20260810 14:35:32.643081 24226 raft_consensus.cc:2229] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:32.643059 23573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:35:32.643234 24226 raft_consensus.cc:2258] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 14:35:32.643569 23573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:32.643777 24224 ts_tablet_manager.cc:1905] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.643784 24225 ts_tablet_manager.cc:1905] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.644244 24226 ts_tablet_manager.cc:1905] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:32.646600 24224 ts_tablet_manager.cc:1918] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:32.646621 24225 ts_tablet_manager.cc:1918] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:32.646688 24224 log.cc:1196] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/eb75dfbb7c294a72bf8989083e8f32ff
I20260810 14:35:32.646701 24225 log.cc:1196] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/eb75dfbb7c294a72bf8989083e8f32ff
I20260810 14:35:32.646986 24225 ts_tablet_manager.cc:1939] T eb75dfbb7c294a72bf8989083e8f32ff P 4f70725e4e7c4795987fd221b6bde4ab: Deleting consensus metadata
I20260810 14:35:32.646987 24224 ts_tablet_manager.cc:1939] T eb75dfbb7c294a72bf8989083e8f32ff P bce61a92bba04aab86519939b6cdf59a: Deleting consensus metadata
I20260810 14:35:32.646955 24226 ts_tablet_manager.cc:1918] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:35:32.647061 24226 log.cc:1196] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/eb75dfbb7c294a72bf8989083e8f32ff
I20260810 14:35:32.647357 24226 ts_tablet_manager.cc:1939] T eb75dfbb7c294a72bf8989083e8f32ff P c87cca4849894dadb6bda662013bf08d: Deleting consensus metadata
I20260810 14:35:32.647872 23558 catalog_manager.cc:4841] TS 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507): tablet eb75dfbb7c294a72bf8989083e8f32ff (table test-table [id=cfe22355a7be47e7a940c2194236e343]) successfully deleted
I20260810 14:35:32.648069 23560 catalog_manager.cc:4841] TS bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703): tablet eb75dfbb7c294a72bf8989083e8f32ff (table test-table [id=cfe22355a7be47e7a940c2194236e343]) successfully deleted
I20260810 14:35:32.648765 23556 catalog_manager.cc:4841] TS c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123): tablet eb75dfbb7c294a72bf8989083e8f32ff (table test-table [id=cfe22355a7be47e7a940c2194236e343]) successfully deleted
I20260810 14:35:32.649168 23722 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8617afc644077946017dc6ad1c739 (DEFAULT_TABLE table=test-table [id=b28b264440c94a1a96d66d6f569e9abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.649184 23875 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8617afc644077946017dc6ad1c739 (DEFAULT_TABLE table=test-table [id=b28b264440c94a1a96d66d6f569e9abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.649214 24036 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf8617afc644077946017dc6ad1c739 (DEFAULT_TABLE table=test-table [id=b28b264440c94a1a96d66d6f569e9abf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:32.649452 23722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8617afc644077946017dc6ad1c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.649454 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8617afc644077946017dc6ad1c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.649454 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf8617afc644077946017dc6ad1c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:32.651371 24236 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: Bootstrap starting.
I20260810 14:35:32.651831 24237 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Bootstrap starting.
I20260810 14:35:32.652302 24236 tablet_bootstrap.cc:654] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.652658 24237 tablet_bootstrap.cc:654] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.652913 24239 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Bootstrap starting.
I20260810 14:35:32.654016 24237 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: No bootstrap required, opened a new log
I20260810 14:35:32.654114 24239 tablet_bootstrap.cc:654] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:32.654134 24237 ts_tablet_manager.cc:1397] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:32.654592 24237 raft_consensus.cc:348] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.654749 24237 raft_consensus.cc:374] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.654805 24237 raft_consensus.cc:729] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.654963 24237 consensus_queue.cc:260] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.655112 24236 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: No bootstrap required, opened a new log
I20260810 14:35:32.655308 24237 ts_tablet_manager.cc:1428] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:35:32.655606 24239 tablet_bootstrap.cc:492] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: No bootstrap required, opened a new log
I20260810 14:35:32.655719 24239 ts_tablet_manager.cc:1397] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:32.656204 24239 raft_consensus.cc:348] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.656328 24239 raft_consensus.cc:374] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.656366 24239 raft_consensus.cc:729] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87cca4849894dadb6bda662013bf08d, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.656509 24239 consensus_queue.cc:260] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.656818 24239 ts_tablet_manager.cc:1428] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:32.657022 24236 ts_tablet_manager.cc:1397] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260810 14:35:32.657543 24236 raft_consensus.cc:348] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.657681 24236 raft_consensus.cc:374] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:32.657737 24236 raft_consensus.cc:729] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce61a92bba04aab86519939b6cdf59a, State: Initialized, Role: FOLLOWER
I20260810 14:35:32.657881 24236 consensus_queue.cc:260] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.658206 24236 ts_tablet_manager.cc:1428] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:35:32.857841 24243 raft_consensus.cc:482] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:32.858022 24243 raft_consensus.cc:504] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.858475 24243 leader_election.cc:288] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:32.859180 23744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8617afc644077946017dc6ad1c739" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61a92bba04aab86519939b6cdf59a" is_pre_election: true
I20260810 14:35:32.859213 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8617afc644077946017dc6ad1c739" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d" is_pre_election: true
I20260810 14:35:32.859375 23744 raft_consensus.cc:2454] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 0.
I20260810 14:35:32.859387 24051 raft_consensus.cc:2454] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 0.
I20260810 14:35:32.859730 23823 leader_election.cc:302] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, c87cca4849894dadb6bda662013bf08d; no voters:
I20260810 14:35:32.859938 24243 raft_consensus.cc:2781] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:32.860020 24243 raft_consensus.cc:482] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:32.860083 24243 raft_consensus.cc:3037] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.861191 24243 raft_consensus.cc:504] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.861601 24243 leader_election.cc:288] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 election: Requested vote from peers bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703), c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123)
I20260810 14:35:32.862215 23744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8617afc644077946017dc6ad1c739" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce61a92bba04aab86519939b6cdf59a"
I20260810 14:35:32.862229 24051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf8617afc644077946017dc6ad1c739" candidate_uuid: "4f70725e4e7c4795987fd221b6bde4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87cca4849894dadb6bda662013bf08d"
I20260810 14:35:32.862357 23744 raft_consensus.cc:3037] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.862401 24051 raft_consensus.cc:3037] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:32.863675 23744 raft_consensus.cc:2454] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 1.
I20260810 14:35:32.863713 24051 raft_consensus.cc:2454] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f70725e4e7c4795987fd221b6bde4ab in term 1.
I20260810 14:35:32.864051 23827 leader_election.cc:302] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70725e4e7c4795987fd221b6bde4ab, bce61a92bba04aab86519939b6cdf59a; no voters:
I20260810 14:35:32.864302 24243 raft_consensus.cc:2781] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:32.864527 24243 raft_consensus.cc:686] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 LEADER]: Becoming Leader. State: Replica: 4f70725e4e7c4795987fd221b6bde4ab, State: Running, Role: LEADER
I20260810 14:35:32.864703 24243 consensus_queue.cc:237] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } }
I20260810 14:35:32.866782 23573 catalog_manager.cc:5495] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66). New cstate: current_term: 1 leader_uuid: "4f70725e4e7c4795987fd221b6bde4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f70725e4e7c4795987fd221b6bde4ab" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:32.911394 24259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:32.907662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.21.43.66:44507)
0810 14:35:32.907990 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:32.907998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:32.908012 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:35:32.908058 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:32.908060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:32.908105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:32.908200 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:32.908480 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.909032 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.910043 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:32.910297 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:32.910390 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:32.910742 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:32.910850 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:32.910901 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:32.911038 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 14:35:32.911096 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":144,"threads_started":1}
I20260810 14:35:32.918406 24051 raft_consensus.cc:1261] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Refusing update from remote peer 4f70725e4e7c4795987fd221b6bde4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:32.918574 23744 raft_consensus.cc:1261] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Refusing update from remote peer 4f70725e4e7c4795987fd221b6bde4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:32.918985 24257 consensus_queue.cc:1035] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87cca4849894dadb6bda662013bf08d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:32.919193 24243 consensus_queue.cc:1035] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce61a92bba04aab86519939b6cdf59a" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:35:32.933573 24206 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:35:34.500054 23571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:34.500239 23571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:34.503093 23571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b048d9d54d4408ebe2732f41d0f01a2: Sending DeleteTablet for 3 replicas of tablet 8bf8617afc644077946017dc6ad1c739
I20260810 14:35:34.503890 23722 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf8617afc644077946017dc6ad1c739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:34 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41572
I20260810 14:35:34.503945 23875 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf8617afc644077946017dc6ad1c739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:34 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:56858
I20260810 14:35:34.503944 24036 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf8617afc644077946017dc6ad1c739 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:34 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:44042
I20260810 14:35:34.504302 24288 tablet_replica.cc:330] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a: stopping tablet replica
I20260810 14:35:34.504467 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23653
I20260810 14:35:34.504474 24288 raft_consensus.cc:2229] T 8bf8617afc644077946017dc6ad1c739 P bce61a92bba04aab86519939b6cdf59a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:34.506599 24289 tablet_replica.cc:330] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: stopping tablet replica
I20260810 14:35:34.506824 24289 raft_consensus.cc:2229] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:34.506989 24289 raft_consensus.cc:2258] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:34.507333 24291 tablet_replica.cc:330] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: stopping tablet replica
I20260810 14:35:34.507499 24291 raft_consensus.cc:2229] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:34.507759 24291 raft_consensus.cc:2258] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:34.508441 24289 ts_tablet_manager.cc:1905] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:34.510931 24289 ts_tablet_manager.cc:1918] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:35:34.511039 24289 log.cc:1196] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/8bf8617afc644077946017dc6ad1c739
I20260810 14:35:34.511377 24289 ts_tablet_manager.cc:1939] T 8bf8617afc644077946017dc6ad1c739 P c87cca4849894dadb6bda662013bf08d: Deleting consensus metadata
I20260810 14:35:34.512208 23556 catalog_manager.cc:4841] TS c87cca4849894dadb6bda662013bf08d (127.21.43.67:39123): tablet 8bf8617afc644077946017dc6ad1c739 (table test-table [id=b28b264440c94a1a96d66d6f569e9abf]) successfully deleted
I20260810 14:35:34.512538 24291 ts_tablet_manager.cc:1905] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:34.514700 24291 ts_tablet_manager.cc:1918] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:35:34.514770 24291 log.cc:1196] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/8bf8617afc644077946017dc6ad1c739
I20260810 14:35:34.515026 24291 ts_tablet_manager.cc:1939] T 8bf8617afc644077946017dc6ad1c739 P 4f70725e4e7c4795987fd221b6bde4ab: Deleting consensus metadata
W20260810 14:35:34.516069 23560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:39703 (error 108)
I20260810 14:35:34.516234 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23805
I20260810 14:35:34.516259 23558 catalog_manager.cc:4841] TS 4f70725e4e7c4795987fd221b6bde4ab (127.21.43.66:44507): tablet 8bf8617afc644077946017dc6ad1c739 (table test-table [id=b28b264440c94a1a96d66d6f569e9abf]) successfully deleted
I20260810 14:35:34.519681 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:34.518990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39703 (local address 127.0.0.1:41598)
0810 14:35:34.519414 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:34.519518 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":188,"threads_started":1}
W20260810 14:35:34.519865 23560 catalog_manager.cc:4584] TS bce61a92bba04aab86519939b6cdf59a (127.21.43.65:39703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bf8617afc644077946017dc6ad1c739: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39703: connect: Connection refused (error 111)
I20260810 14:35:34.528023 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23958
I20260810 14:35:34.537649 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 23539
2026-08-10T14:35:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8522 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:34 dist-test-slave-chsk krb5kdc[24296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:34 dist-test-slave-chsk krb5kdc[24296](info): set up 2 sockets
Aug 10 14:35:34 dist-test-slave-chsk krb5kdc[24296](info): commencing operation
krb5kdc: starting...
W20260810 14:35:35.652400 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:35 dist-test-slave-chsk krb5kdc[24296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:35.796458 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35769
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:35.914755 24311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:35.915117 24311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:35.915222 24311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:35.919366 24311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:35.919484 24311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:35.919518 24311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:35.919549 24311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:35.919600 24311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:35.925285 24311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:35.926507 24311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:35.927645 24311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:35.936952 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:35.936996 24311 server_base.cc:1034] running on GCE node
W20260810 14:35:35.936942 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:35.936942 24320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:35.937834 24311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:35.938995 24311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:35.940207 24311 hybrid_clock.cc:648] HybridClock initialized: now 1786372535940188 us; error 36 us; skew 500 ppm
Aug 10 14:35:35 dist-test-slave-chsk krb5kdc[24296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372535, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:35.943104 24311 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:35.944099 24311 webserver.cc:460] Webserver started at http://127.21.43.126:35169/ using document root <none> and password file <none>
I20260810 14:35:35.944684 24311 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:35.944748 24311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:35.944975 24311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:35.947065 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "098022c45a30489cba08b1db3b3632d0"
format_stamp: "Formatted at 2026-08-10 14:35:35 on dist-test-slave-chsk"
server_key: "c4927be9e304ddee66847f277580c910"
server_key_iv: "106e7977393079fc5c7fd244681314f6"
server_key_version: "encryptionkey@0"
I20260810 14:35:35.947721 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "098022c45a30489cba08b1db3b3632d0"
format_stamp: "Formatted at 2026-08-10 14:35:35 on dist-test-slave-chsk"
server_key: "c4927be9e304ddee66847f277580c910"
server_key_iv: "106e7977393079fc5c7fd244681314f6"
server_key_version: "encryptionkey@0"
I20260810 14:35:35.951511 24311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:35.954016 24329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:35.955403 24311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:35.955551 24311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "098022c45a30489cba08b1db3b3632d0"
format_stamp: "Formatted at 2026-08-10 14:35:35 on dist-test-slave-chsk"
server_key: "c4927be9e304ddee66847f277580c910"
server_key_iv: "106e7977393079fc5c7fd244681314f6"
server_key_version: "encryptionkey@0"
I20260810 14:35:35.955669 24311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:35.982482 24311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:35.985597 24311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:35.985837 24311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:35.993687 24311 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:35769
I20260810 14:35:35.993708 24384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:35769 every 8 connection(s)
I20260810 14:35:35.994803 24311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:35.997812 24385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:36.004055 24385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0: Bootstrap starting.
I20260810 14:35:36.005036 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 24311
I20260810 14:35:36.005155 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:36.005457 21677 external_mini_cluster.cc:1428] Setting key eeb851c3c92ef7c44cae550d5faae33a
I20260810 14:35:36.007470 24385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:36.008311 24385 log.cc:824] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:36.010358 24385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0: No bootstrap required, opened a new log
I20260810 14:35:36.014686 24385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } }
I20260810 14:35:36.015062 24385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:36.015214 24385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098022c45a30489cba08b1db3b3632d0, State: Initialized, Role: FOLLOWER
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:36.015836 24385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } }
I20260810 14:35:36.016034 24385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:36.016141 24385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:36.016260 24385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:36.017869 24385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } }
I20260810 14:35:36.018296 24385 leader_election.cc:302] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 098022c45a30489cba08b1db3b3632d0; no voters:
I20260810 14:35:36.018759 24385 leader_election.cc:288] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:36.018900 24390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:36.019192 24390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [term 1 LEADER]: Becoming Leader. State: Replica: 098022c45a30489cba08b1db3b3632d0, State: Running, Role: LEADER
I20260810 14:35:36.019557 24390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } }
I20260810 14:35:36.020359 24385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:36.021569 24392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 098022c45a30489cba08b1db3b3632d0. Latest consensus state: current_term: 1 leader_uuid: "098022c45a30489cba08b1db3b3632d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } } }
I20260810 14:35:36.021711 24392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:36.022027 24391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "098022c45a30489cba08b1db3b3632d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "098022c45a30489cba08b1db3b3632d0" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35769 } } }
I20260810 14:35:36.022130 24391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:36.022251 24401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:36.022534 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.006836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.21.43.126:35769)
0810 14:35:36.007377 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:36.007393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:36.007431 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:35:36.008227 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:36.008245 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:36.008659 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:36.009007 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:36.009919 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.010986 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.011933 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.012471 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.012671 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.016199 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:36.016206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:36.016226 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:36.016269 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:36.018547 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.019198 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.019205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.019212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.019362 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.019722 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.019725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.019727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.020276 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:36.020481 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:36.021005 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:35:36.021387 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":151,"threads_started":1}
I20260810 14:35:36.026264 24401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:36.032723 24401 catalog_manager.cc:1347] Generated new cluster ID: a1274bec45394dbf977b34905857566a
I20260810 14:35:36.032821 24401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:36.044097 24401 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:36.045120 24401 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:36.056134 24401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 098022c45a30489cba08b1db3b3632d0: Generated new TSK 0
I20260810 14:35:36.057184 24401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:36.100876 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:36.218743 24415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:36.219038 24415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:36.219086 24415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:36.223706 24415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:36.223802 24415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:36.223868 24415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:36.229780 24415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:36.230911 24415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:36.231959 24415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:35:36.242261 24415 server_base.cc:1034] running on GCE node
W20260810 14:35:36.242188 24420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.242188 24421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.242189 24423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:36.242825 24415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:36.243614 24415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:36.244817 24415 hybrid_clock.cc:648] HybridClock initialized: now 1786372536244781 us; error 50 us; skew 500 ppm
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:36.247927 24415 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:36.248916 24415 webserver.cc:460] Webserver started at http://127.21.43.65:33283/ using document root <none> and password file <none>
I20260810 14:35:36.249526 24415 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:36.249621 24415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:36.249871 24415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:36.252076 24415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "8aa57ad10e8a47adbeef61cfe5a6dae7"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "666f8812787f830d79a95afad33c2d15"
server_key_iv: "7fea53e3cd7c3b869a8f76f9bd8923bb"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.252679 24415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "8aa57ad10e8a47adbeef61cfe5a6dae7"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "666f8812787f830d79a95afad33c2d15"
server_key_iv: "7fea53e3cd7c3b869a8f76f9bd8923bb"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.256561 24415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:36.259479 24430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.260865 24415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:36.261009 24415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "8aa57ad10e8a47adbeef61cfe5a6dae7"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "666f8812787f830d79a95afad33c2d15"
server_key_iv: "7fea53e3cd7c3b869a8f76f9bd8923bb"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.261137 24415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:36.281704 24415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:36.285043 24415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:36.285284 24415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:36.285936 24415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:36.286829 24415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:36.286913 24415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.286988 24415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:36.287024 24415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.297233 24415 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39409
I20260810 14:35:36.297272 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39409 every 8 connection(s)
I20260810 14:35:36.298231 24415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:36.298885 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 24415
I20260810 14:35:36.299012 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:36.299371 21677 external_mini_cluster.cc:1428] Setting key 4c45a2385255a927538370d0f916073f
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:36.316198 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.300801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:54971 (local address 127.21.43.126:35769)
0810 14:35:36.300947 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:36.300951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:36.301688 (+ 737us) server_negotiation.cc:407] Connection header received
0810 14:35:36.303862 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:36.303865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:36.303966 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:36.304116 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:36.305565 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.306117 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.308156 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.308402 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.308497 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.311883 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:36.311887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:36.311889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:36.311918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:36.313468 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.314105 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.314110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.314112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.314213 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.314740 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.314743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.314744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.315015 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:36.315100 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:36.315745 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 14:35:36.315918 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:36.316927 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.300882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35769 (local address 127.21.43.65:54971)
0810 14:35:36.301519 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:36.301567 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:36.303559 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:36.304308 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:36.304316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:36.304719 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:36.305278 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.305303 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.306288 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.306295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.307953 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.307970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.308594 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.308602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.308860 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.309569 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:36.309595 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:36.311656 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:36.313601 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.313607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.313621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.313914 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.314395 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.314398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.314399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.314566 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.315161 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:36.315171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:36.315457 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 14:35:36.315633 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 14:35:36.315933 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":187,"threads_started":1}
I20260810 14:35:36.318397 24556 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35769
I20260810 14:35:36.318702 24556 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:36.319309 24556 heartbeater.cc:511] Master 127.21.43.126:35769 requested a full tablet report, sending...
I20260810 14:35:36.321101 24348 ts_manager.cc:194] Registered new tserver with Master: 8aa57ad10e8a47adbeef61cfe5a6dae7 (127.21.43.65:39409)
I20260810 14:35:36.322472 24348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:54971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:36.334551 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:36.458997 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:36.459702 24561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:36.459839 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:36.464303 24561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:36.464411 24561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:36.464501 24561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:36.470353 24561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:36.471581 24561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:36.472635 24561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:36.482630 24569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.482620 24567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.482671 24571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:36.482873 24561 server_base.cc:1034] running on GCE node
I20260810 14:35:36.483486 24561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:36.484223 24561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:36.485405 24561 hybrid_clock.cc:648] HybridClock initialized: now 1786372536485378 us; error 45 us; skew 500 ppm
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:36.488552 24561 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:36.489504 24561 webserver.cc:460] Webserver started at http://127.21.43.66:33539/ using document root <none> and password file <none>
I20260810 14:35:36.490077 24561 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:36.490146 24561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:36.490324 24561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:36.492583 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "f6a09c418bf74dc588852586db0f7a01"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "f638eb10ea4e2a21a4c49705ebaa371d"
server_key_iv: "2d22ea78551f1538514a7dde6bac27a0"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.493172 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "f6a09c418bf74dc588852586db0f7a01"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "f638eb10ea4e2a21a4c49705ebaa371d"
server_key_iv: "2d22ea78551f1538514a7dde6bac27a0"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.497342 24561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:36.500119 24578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.501505 24561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:36.501655 24561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "f6a09c418bf74dc588852586db0f7a01"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "f638eb10ea4e2a21a4c49705ebaa371d"
server_key_iv: "2d22ea78551f1538514a7dde6bac27a0"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.501791 24561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:36.530817 24561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:36.534704 24561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:36.534931 24561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:36.535679 24561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:36.536578 24561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:36.536646 24561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.536720 24561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:36.536756 24561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.546207 24561 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:41759
I20260810 14:35:36.546224 24701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:41759 every 8 connection(s)
I20260810 14:35:36.547194 24561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:36.552970 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 24561
I20260810 14:35:36.553088 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:36.553359 21677 external_mini_cluster.cc:1428] Setting key dc12c13ac064000b8eeebd2fc1801d37
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:36.561887 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.549045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:44819 (local address 127.21.43.126:35769)
0810 14:35:36.549211 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:36.549216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:36.550109 (+ 893us) server_negotiation.cc:407] Connection header received
0810 14:35:36.551453 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:36.551456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:36.551504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:36.551579 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:36.552683 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.553511 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.554783 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.555076 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.555247 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.557809 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:36.557812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:36.557814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:36.557837 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:36.559468 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.560153 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.560156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.560157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.560246 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.560676 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.560678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.560679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.560866 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:36.560946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:36.561654 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 14:35:36.561758 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:35:36.563164 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.549490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35769 (local address 127.21.43.66:44819)
0810 14:35:36.549943 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:36.549984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:36.551261 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:36.551743 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:36.551749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:36.552093 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:36.552554 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.552567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.553642 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.553645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.554641 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.554656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.555297 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.555301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.555439 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.556063 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:36.556080 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:36.557634 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:36.559593 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.559602 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.559621 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.560021 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.560358 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.560361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.560363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.560567 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.560982 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:36.560992 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:36.561366 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 14:35:36.561608 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 14:35:36.562015 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260810 14:35:36.564370 24702 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35769
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:35:36.564710 24702 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:36.565197 24702 heartbeater.cc:511] Master 127.21.43.126:35769 requested a full tablet report, sending...
I20260810 14:35:36.566308 24348 ts_manager.cc:194] Registered new tserver with Master: f6a09c418bf74dc588852586db0f7a01 (127.21.43.66:41759)
I20260810 14:35:36.566840 24348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:44819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:36.583855 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:36.702016 24707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:36.702369 24707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:36.702450 24707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:36.706881 24707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:36.706991 24707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:36.707091 24707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:36.712944 24707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:36.714216 24707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:36.715396 24707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:35:36.726359 24707 server_base.cc:1034] running on GCE node
W20260810 14:35:36.726315 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.726318 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:36.726315 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:36.727075 24707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:36.727837 24707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:36.729063 24707 hybrid_clock.cc:648] HybridClock initialized: now 1786372536729044 us; error 47 us; skew 500 ppm
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:36.732275 24707 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:36.733287 24707 webserver.cc:460] Webserver started at http://127.21.43.67:43303/ using document root <none> and password file <none>
I20260810 14:35:36.734151 24707 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:36.734302 24707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:36.734572 24707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:36.737071 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "0225149719f04c10a6aea5c58f445816"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "697e4ac88a23eb47c734da64a17a5885"
server_key_iv: "618c4eac5940abd123f15e905a7cab6b"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.737716 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "0225149719f04c10a6aea5c58f445816"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "697e4ac88a23eb47c734da64a17a5885"
server_key_iv: "618c4eac5940abd123f15e905a7cab6b"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.742027 24707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:36.744782 24722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.746188 24707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:36.746371 24707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "0225149719f04c10a6aea5c58f445816"
format_stamp: "Formatted at 2026-08-10 14:35:36 on dist-test-slave-chsk"
server_key: "697e4ac88a23eb47c734da64a17a5885"
server_key_iv: "618c4eac5940abd123f15e905a7cab6b"
server_key_version: "encryptionkey@0"
I20260810 14:35:36.746510 24707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:36.759358 24707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:36.763023 24707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:36.763300 24707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:36.763981 24707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:36.764958 24707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:36.765043 24707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.765136 24707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:36.765182 24707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:36.775456 24707 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:45617
I20260810 14:35:36.775475 24843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:45617 every 8 connection(s)
I20260810 14:35:36.776453 24707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:36.781594 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 24707
I20260810 14:35:36.781759 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:36.782063 21677 external_mini_cluster.cc:1428] Setting key 435460e2a009c16ded1ef04e8b5072af
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372536, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:36.791318 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.778387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:58501 (local address 127.21.43.126:35769)
0810 14:35:36.778567 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:36.778571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:36.779494 (+ 923us) server_negotiation.cc:407] Connection header received
0810 14:35:36.780867 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:36.780871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:36.780934 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:36.781050 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:36.782461 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.783249 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.784787 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.785040 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.785162 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.787846 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:36.787849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:36.787851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:36.787879 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:36.789138 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.789657 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.789659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.789660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.789740 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:36.790091 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:36.790094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:36.790095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:36.790315 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:36.790408 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:36.790941 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 14:35:36.791112 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 14:35:36.792181 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:36.778776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35769 (local address 127.21.43.67:58501)
0810 14:35:36.779342 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:36.779388 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:36.780639 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:36.781238 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:36.781245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:36.781643 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:36.782286 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.782302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.783384 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.783388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.784625 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:36.784636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:36.785223 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:36.785228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:36.785407 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:36.786008 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:36.786026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:36.787656 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:36.789256 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.789261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.789274 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.789550 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.789825 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:36.789827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:36.789828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:36.789966 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:36.790417 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:36.790422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:36.790688 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 14:35:36.790854 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 14:35:36.791229 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260810 14:35:36.793466 24844 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35769
I20260810 14:35:36.793737 24844 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:36.794302 24844 heartbeater.cc:511] Master 127.21.43.126:35769 requested a full tablet report, sending...
I20260810 14:35:36.795531 24348 ts_manager.cc:194] Registered new tserver with Master: 0225149719f04c10a6aea5c58f445816 (127.21.43.67:45617)
I20260810 14:35:36.796309 24348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:58501
I20260810 14:35:36.796413 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:35:36.813241 24388 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:36.811691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.21.43.126:35769)
0810 14:35:36.811832 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:36.811835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:36.811998 (+ 163us) server_negotiation.cc:407] Connection header received
0810 14:35:36.812148 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:36.812150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:36.812197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:36.812282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:36.813050 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59384: BlockingRecv error: recv got EOF from 127.0.0.1:59384 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:36.814649 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 24415
I20260810 14:35:36.823323 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 24561
I20260810 14:35:36.833312 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 24707
I20260810 14:35:36.841187 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 24311
2026-08-10T14:35:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2302 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24857](info): set up 2 sockets
Aug 10 14:35:36 dist-test-slave-chsk krb5kdc[24857](info): commencing operation
krb5kdc: starting...
W20260810 14:35:37.954249 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:38.086944 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41475
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:38.198635 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:38.198939 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:38.199026 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:38.203114 24871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:38.203230 24871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:38.203269 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:38.203305 24871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:38.203339 24871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:38.209167 24871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:38.210296 24871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:38.211407 24871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:38.219595 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:38.219646 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:38.219746 24871 server_base.cc:1034] running on GCE node
W20260810 14:35:38.219631 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:38.220417 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:38.221485 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:38.222683 24871 hybrid_clock.cc:648] HybridClock initialized: now 1786372538222661 us; error 37 us; skew 500 ppm
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372538, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:38.225441 24871 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:38.226382 24871 webserver.cc:460] Webserver started at http://127.21.43.126:34691/ using document root <none> and password file <none>
I20260810 14:35:38.226945 24871 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:38.227039 24871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:38.227321 24871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:38.229377 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "172fbfb2a4014fda88b0c42f453f21bd"
format_stamp: "Formatted at 2026-08-10 14:35:38 on dist-test-slave-chsk"
server_key: "96fb1fa3334229f60a5981a5dd93886e"
server_key_iv: "70577c8ab2beb59d38026cb1bf3e8526"
server_key_version: "encryptionkey@0"
I20260810 14:35:38.229979 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "172fbfb2a4014fda88b0c42f453f21bd"
format_stamp: "Formatted at 2026-08-10 14:35:38 on dist-test-slave-chsk"
server_key: "96fb1fa3334229f60a5981a5dd93886e"
server_key_iv: "70577c8ab2beb59d38026cb1bf3e8526"
server_key_version: "encryptionkey@0"
I20260810 14:35:38.233798 24871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:38.236320 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:38.237524 24871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:38.237675 24871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "172fbfb2a4014fda88b0c42f453f21bd"
format_stamp: "Formatted at 2026-08-10 14:35:38 on dist-test-slave-chsk"
server_key: "96fb1fa3334229f60a5981a5dd93886e"
server_key_iv: "70577c8ab2beb59d38026cb1bf3e8526"
server_key_version: "encryptionkey@0"
I20260810 14:35:38.237788 24871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:38.269951 24871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:38.272977 24871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:38.273193 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:38.280639 24871 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:41475
I20260810 14:35:38.280958 24947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:41475 every 8 connection(s)
I20260810 14:35:38.281646 24871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:38.284264 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 24871
I20260810 14:35:38.284403 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:38.284511 24948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:38.284736 21677 external_mini_cluster.cc:1428] Setting key bcd13589196803dc2073ab8ff7b9a244
I20260810 14:35:38.292065 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd: Bootstrap starting.
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:38.294555 24948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:38.295511 24948 log.cc:824] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:38.297931 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd: No bootstrap required, opened a new log
I20260810 14:35:38.300171 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:38.286149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.21.43.126:41475)
0810 14:35:38.286591 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:38.286609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:38.286652 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:35:38.287411 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:38.287435 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:38.287873 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:38.288273 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:38.289080 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:38.290573 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:38.291567 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:38.292251 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:38.292409 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:38.294919 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:38.294925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:38.294941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:38.294972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:38.297045 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:38.297478 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:38.297484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:38.297489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:38.297626 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:38.297905 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:38.297908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:38.297910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:38.298422 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:38.298611 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:38.298861 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:35:38.299197 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260810 14:35:38.302433 24948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } }
I20260810 14:35:38.302719 24948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:38.302784 24948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172fbfb2a4014fda88b0c42f453f21bd, State: Initialized, Role: FOLLOWER
I20260810 14:35:38.303298 24948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } }
I20260810 14:35:38.303388 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:35:38.303463 24948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:38.303529 24948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:38.303635 24948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:38.304556 24948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } }
I20260810 14:35:38.304911 24948 leader_election.cc:302] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 172fbfb2a4014fda88b0c42f453f21bd; no voters:
I20260810 14:35:38.305251 24948 leader_election.cc:288] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:38.305351 24955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:38.305562 24955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [term 1 LEADER]: Becoming Leader. State: Replica: 172fbfb2a4014fda88b0c42f453f21bd, State: Running, Role: LEADER
I20260810 14:35:38.305925 24955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } }
I20260810 14:35:38.306588 24948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:38.307649 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "172fbfb2a4014fda88b0c42f453f21bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } } }
I20260810 14:35:38.307796 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:38.308197 24957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 172fbfb2a4014fda88b0c42f453f21bd. Latest consensus state: current_term: 1 leader_uuid: "172fbfb2a4014fda88b0c42f453f21bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fbfb2a4014fda88b0c42f453f21bd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41475 } } }
I20260810 14:35:38.308362 24957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:38.309104 24965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:38.312675 24965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:38.318495 24965 catalog_manager.cc:1347] Generated new cluster ID: 137f73b7469a4dd3adee1a50683803c0
I20260810 14:35:38.318616 24965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:38.327265 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:38.321439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.21.43.126:41475)
0810 14:35:38.321543 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:38.321547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:38.321831 (+ 284us) server_negotiation.cc:407] Connection header received
0810 14:35:38.321941 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:38.321944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:38.322018 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:38.322121 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:38.322694 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:38.323560 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:38.324734 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:38.325104 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:38.325216 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:38.326156 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:38.326160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:38.326163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:38.326752 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:38.326788 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:38.326808 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:38.326902 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:38.327005 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:35:38.327083 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:35:38.330603 24965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:38.331872 24965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:38.337812 24965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 172fbfb2a4014fda88b0c42f453f21bd: Generated new TSK 0
I20260810 14:35:38.338596 24965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:35:38.344691 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 24871
2026-08-10T14:35:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1500 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24986](info): set up 2 sockets
Aug 10 14:35:38 dist-test-slave-chsk krb5kdc[24986](info): commencing operation
krb5kdc: starting...
W20260810 14:35:39.442281 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:39 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:39.574934 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35263
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:39.691435 25000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:39.691715 25000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:39.691756 25000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:39.695803 25000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:39.695871 25000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:39.695890 25000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:39.695906 25000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:39.695921 25000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:39.701330 25000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:39.702574 25000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:39.703773 25000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:39.712670 25008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:39.712663 25005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:39.712678 25006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:39.713833 25000 server_base.cc:1034] running on GCE node
I20260810 14:35:39.714473 25000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:39.715721 25000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:39.716921 25000 hybrid_clock.cc:648] HybridClock initialized: now 1786372539716901 us; error 31 us; skew 500 ppm
Aug 10 14:35:39 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372539, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:39.723459 25000 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:39.724515 25000 webserver.cc:460] Webserver started at http://127.21.43.126:40693/ using document root <none> and password file <none>
I20260810 14:35:39.725117 25000 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:39.725185 25000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:39.725440 25000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:39.727634 25000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "55b60618e0364846a0009db87050dbb4"
format_stamp: "Formatted at 2026-08-10 14:35:39 on dist-test-slave-chsk"
server_key: "7c0799cac2645ab3b927605706dbfc1e"
server_key_iv: "b18fd2d548a1ac615d417cd33209168b"
server_key_version: "encryptionkey@0"
I20260810 14:35:39.728272 25000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "55b60618e0364846a0009db87050dbb4"
format_stamp: "Formatted at 2026-08-10 14:35:39 on dist-test-slave-chsk"
server_key: "7c0799cac2645ab3b927605706dbfc1e"
server_key_iv: "b18fd2d548a1ac615d417cd33209168b"
server_key_version: "encryptionkey@0"
I20260810 14:35:39.732347 25000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:39.735651 25016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:39.737265 25000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:35:39.737413 25000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "55b60618e0364846a0009db87050dbb4"
format_stamp: "Formatted at 2026-08-10 14:35:39 on dist-test-slave-chsk"
server_key: "7c0799cac2645ab3b927605706dbfc1e"
server_key_iv: "b18fd2d548a1ac615d417cd33209168b"
server_key_version: "encryptionkey@0"
I20260810 14:35:39.737550 25000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:39.763849 25000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:39.767040 25000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:39.767295 25000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:39.774665 25000 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:35263
I20260810 14:35:39.774962 25076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:35263 every 8 connection(s)
I20260810 14:35:39.775748 25000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:39.778651 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:39.783375 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25000
I20260810 14:35:39.783486 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:39.783761 21677 external_mini_cluster.cc:1428] Setting key 562db3e0e84e7099930d4a7d2cf1d634
I20260810 14:35:39.784548 25077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: Bootstrap starting.
I20260810 14:35:39.787114 25077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:39.788105 25077 log.cc:824] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:39.790421 25077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: No bootstrap required, opened a new log
Aug 10 14:35:39 dist-test-slave-chsk krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:39.795071 25077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } }
I20260810 14:35:39.795385 25077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:39.795516 25077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b60618e0364846a0009db87050dbb4, State: Initialized, Role: FOLLOWER
I20260810 14:35:39.796010 25077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } }
I20260810 14:35:39.796177 25077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:39.796272 25077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:39.796401 25077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:39.797387 25077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } }
I20260810 14:35:39.797806 25077 leader_election.cc:302] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b60618e0364846a0009db87050dbb4; no voters:
I20260810 14:35:39.798206 25077 leader_election.cc:288] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:39.798317 25082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:39.798625 25082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [term 1 LEADER]: Becoming Leader. State: Replica: 55b60618e0364846a0009db87050dbb4, State: Running, Role: LEADER
I20260810 14:35:39.799088 25082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } }
I20260810 14:35:39.799681 25077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:39.800457 25082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b60618e0364846a0009db87050dbb4. Latest consensus state: current_term: 1 leader_uuid: "55b60618e0364846a0009db87050dbb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } } }
I20260810 14:35:39.800601 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:39.785163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.21.43.126:35263)
0810 14:35:39.785594 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:39.785611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:39.785649 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:35:39.786293 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:39.786311 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:39.786650 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:39.786935 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:39.787999 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:39.789264 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:39.790434 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:39.791107 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:39.791351 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:39.794338 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:39.794344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:39.794360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:39.794396 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:39.796627 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:39.797119 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:39.797126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:39.797132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:39.797265 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:39.797584 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:39.797588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:39.797590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:39.798174 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:39.798382 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:39.799266 (+ 884us) server_negotiation.cc:299] Negotiation successful
0810 14:35:39.799603 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260810 14:35:39.800927 25083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b60618e0364846a0009db87050dbb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b60618e0364846a0009db87050dbb4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35263 } } }
I20260810 14:35:39.802042 25083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:39.800637 25082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4 [sys.catalog]: This master's current role is: LEADER
W20260810 14:35:39.805013 25096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:35:39.805110 25096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:35:39.805205 25097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:39.805929 25097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:39.813534 25097 catalog_manager.cc:1347] Generated new cluster ID: 113c442d857442d684efde5ffd9f3715
I20260810 14:35:39.813632 25097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:39.828750 25097 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:39.832129 25097 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:39.843222 25097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55b60618e0364846a0009db87050dbb4: Generated new TSK 0
I20260810 14:35:39.844138 25097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:39.880350 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:39.998574 25104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:39.998914 25104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:39.998996 25104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:40.003583 25104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:40.003682 25104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:40.003777 25104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:40.009343 25104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:40.010541 25104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:40.011662 25104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:40.022324 25112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:40.022360 25104 server_base.cc:1034] running on GCE node
W20260810 14:35:40.022324 25109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:40.022395 25110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:40.023041 25104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:40.023821 25104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:40.024991 25104 hybrid_clock.cc:648] HybridClock initialized: now 1786372540024979 us; error 27 us; skew 500 ppm
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:40.028036 25104 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:40.028990 25104 webserver.cc:460] Webserver started at http://127.21.43.65:35061/ using document root <none> and password file <none>
I20260810 14:35:40.029740 25104 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:40.029841 25104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:40.030099 25104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:40.032970 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "f641a1b183b54bebbbfc542283e25d67"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "0fc52ef103617c0f9c2e545c64e433f5"
server_key_iv: "4c9a5d9ba0ea055e3b20ad9e8cb7e775"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.033578 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "f641a1b183b54bebbbfc542283e25d67"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "0fc52ef103617c0f9c2e545c64e433f5"
server_key_iv: "4c9a5d9ba0ea055e3b20ad9e8cb7e775"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.037505 25104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:40.040158 25121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.041517 25104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:40.041693 25104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "f641a1b183b54bebbbfc542283e25d67"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "0fc52ef103617c0f9c2e545c64e433f5"
server_key_iv: "4c9a5d9ba0ea055e3b20ad9e8cb7e775"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.041817 25104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:40.077324 25104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:40.080654 25104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:40.080875 25104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:40.081530 25104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:40.082427 25104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:40.082501 25104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.082576 25104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:40.082626 25104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.091982 25104 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:43557
I20260810 14:35:40.092010 25237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:43557 every 8 connection(s)
I20260810 14:35:40.092998 25104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:40.098798 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25104
I20260810 14:35:40.098955 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:40.099325 21677 external_mini_cluster.cc:1428] Setting key 25ef04db294b5625b6047e764ece19df
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:40.109637 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.095047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:39679 (local address 127.21.43.126:35263)
0810 14:35:40.095235 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:40.095239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:40.096193 (+ 954us) server_negotiation.cc:407] Connection header received
0810 14:35:40.097630 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:40.097634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:40.097695 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:40.097808 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:40.099066 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.099865 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.101442 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.101831 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.101963 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.104979 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:40.104982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:40.104984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:40.105012 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:40.106809 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.107649 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.107653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.107654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.107750 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.108235 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.108237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.108238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.108441 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:40.108519 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:40.109338 (+ 819us) server_negotiation.cc:299] Negotiation successful
0810 14:35:40.109486 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:40.110882 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.095484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35263 (local address 127.21.43.65:39679)
0810 14:35:40.096011 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:40.096057 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:40.097401 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:40.097999 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:40.098006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:40.098388 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:40.098875 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.098888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.100065 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.100068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.101306 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.101324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.102023 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.102027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.102181 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.102852 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:40.102870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:40.104749 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:40.106969 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.106981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.107004 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.107512 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.107860 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.107865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.107870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.108119 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.108565 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:40.108575 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:40.108979 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 14:35:40.109225 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 14:35:40.109651 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260810 14:35:40.112361 25238 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35263
I20260810 14:35:40.112598 25238 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:40.113174 25238 heartbeater.cc:511] Master 127.21.43.126:35263 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:40.115052 25035 ts_manager.cc:194] Registered new tserver with Master: f641a1b183b54bebbbfc542283e25d67 (127.21.43.65:43557)
I20260810 14:35:40.116479 25035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:39679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:40.131099 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:40.246227 25243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:40.246497 25243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:40.246543 25243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:40.250543 25243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:40.250600 25243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:40.250658 25243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:40.255898 25243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:40.256922 25243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:40.257905 25243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:40.267681 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:40.267684 25253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:40.267851 25243 server_base.cc:1034] running on GCE node
W20260810 14:35:40.267715 25249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:40.268383 25243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:40.269080 25243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:40.270287 25243 hybrid_clock.cc:648] HybridClock initialized: now 1786372540270258 us; error 47 us; skew 500 ppm
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:40.273362 25243 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:40.274338 25243 webserver.cc:460] Webserver started at http://127.21.43.66:44607/ using document root <none> and password file <none>
I20260810 14:35:40.274941 25243 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:40.275041 25243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:40.275338 25243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:40.277405 25243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "bc6b1560f4b04783b48447168c00727a"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "dd9155253a0b49e824d0480df22679a0"
server_key_iv: "5b43ab7ba5c5d020c389b4ae88abedf6"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.278060 25243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "bc6b1560f4b04783b48447168c00727a"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "dd9155253a0b49e824d0480df22679a0"
server_key_iv: "5b43ab7ba5c5d020c389b4ae88abedf6"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.281946 25243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:40.284485 25261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.285715 25243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:40.285856 25243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "bc6b1560f4b04783b48447168c00727a"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "dd9155253a0b49e824d0480df22679a0"
server_key_iv: "5b43ab7ba5c5d020c389b4ae88abedf6"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.285974 25243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:40.302552 25243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:40.306000 25243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:40.306164 25243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:40.306718 25243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:40.307595 25243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:40.307642 25243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.307685 25243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:40.307700 25243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.316953 25243 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:38845
I20260810 14:35:40.316979 25385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:38845 every 8 connection(s)
I20260810 14:35:40.317863 25243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:40.318812 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25243
I20260810 14:35:40.318943 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:40.319317 21677 external_mini_cluster.cc:1428] Setting key f7bb7f0f102163c20efa6227d80c538a
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:40.334131 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.319921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:35013 (local address 127.21.43.126:35263)
0810 14:35:40.320082 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:40.320087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:40.321094 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 14:35:40.323166 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:40.323172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:40.323255 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:40.323360 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:40.324519 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.325425 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.326661 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.327111 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.327279 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.330203 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:40.330206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:40.330208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:40.330240 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:40.331885 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.332442 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.332445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.332446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.332524 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.332912 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.332914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.332915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.333149 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:40.333227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:40.333822 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 14:35:40.333957 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:35:40.334971 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.320374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35263 (local address 127.21.43.66:35013)
0810 14:35:40.320926 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:40.320983 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:40.322873 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:40.323548 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:40.323555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:40.323904 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:40.324389 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.324402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.325561 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.325564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.326528 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.326538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.327297 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.327300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.327447 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.328078 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:40.328095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:40.330000 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:40.332003 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.332008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.332022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.332350 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.332616 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.332618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.332620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.332799 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.333275 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:40.333285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:40.333557 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 14:35:40.333776 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 14:35:40.334122 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260810 14:35:40.336222 25386 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35263
I20260810 14:35:40.336483 25386 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:40.336985 25386 heartbeater.cc:511] Master 127.21.43.126:35263 requested a full tablet report, sending...
I20260810 14:35:40.338106 25035 ts_manager.cc:194] Registered new tserver with Master: bc6b1560f4b04783b48447168c00727a (127.21.43.66:38845)
I20260810 14:35:40.338703 25035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:35013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:40.350011 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:40.469575 25391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:40.469873 25391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:40.469955 25391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:40.474628 25391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:40.474727 25391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:40.474848 25391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:40.480748 25391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:40.481933 25391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:40.483060 25391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:40.493511 25396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:40.493511 25397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:40.493505 25399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:40.493938 25391 server_base.cc:1034] running on GCE node
I20260810 14:35:40.494437 25391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:40.495215 25391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:40.496470 25391 hybrid_clock.cc:648] HybridClock initialized: now 1786372540496435 us; error 40 us; skew 500 ppm
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:40.499557 25391 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:40.500666 25391 webserver.cc:460] Webserver started at http://127.21.43.67:44927/ using document root <none> and password file <none>
I20260810 14:35:40.501365 25391 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:40.501436 25391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:40.501622 25391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:40.503793 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "c2f575df909243b3a3d5597ecf113108"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "1e72a1d724c9479af47c37f96a08560a"
server_key_iv: "a3c301ef4fe1a051651ee6203852f2b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.504328 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "c2f575df909243b3a3d5597ecf113108"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "1e72a1d724c9479af47c37f96a08560a"
server_key_iv: "a3c301ef4fe1a051651ee6203852f2b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.508265 25391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:40.510874 25407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.512212 25391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:40.512313 25391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "c2f575df909243b3a3d5597ecf113108"
format_stamp: "Formatted at 2026-08-10 14:35:40 on dist-test-slave-chsk"
server_key: "1e72a1d724c9479af47c37f96a08560a"
server_key_iv: "a3c301ef4fe1a051651ee6203852f2b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:40.512406 25391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:40.557260 25391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:40.560899 25391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:40.561084 25391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:40.561669 25391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:40.562527 25391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:40.562577 25391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.562618 25391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:40.562633 25391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:40.572021 25391 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:41283
I20260810 14:35:40.572057 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:41283 every 8 connection(s)
I20260810 14:35:40.572952 25391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:40.578084 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25391
I20260810 14:35:40.578258 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:40.578550 21677 external_mini_cluster.cc:1428] Setting key 34588bfd0ee36db0de561dd340227c20
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:40.586961 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.574820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:58921 (local address 127.21.43.126:35263)
0810 14:35:40.574967 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:40.574971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:40.575873 (+ 902us) server_negotiation.cc:407] Connection header received
0810 14:35:40.577265 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:40.577268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:40.577329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:40.577437 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:40.578748 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.579481 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.580843 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.581110 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.581201 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.583737 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:40.583740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:40.583743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:40.583768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:40.585078 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.585565 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.585569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.585570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.585650 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.585965 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.585967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.585968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.586172 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:40.586251 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:40.586726 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 14:35:40.586833 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:40.587855 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.575252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35263 (local address 127.21.43.67:58921)
0810 14:35:40.575723 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:40.575765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:40.577046 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:40.577622 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:40.577628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:40.577992 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:40.578568 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.578582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.579617 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.579628 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.580665 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:40.580676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.581258 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.581261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:40.581406 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.582050 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:40.582066 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:40.583578 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:40.585187 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.585192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.585211 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.585469 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.585737 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:40.585739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:40.585740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:40.585883 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:40.586262 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:40.586267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:40.586510 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 14:35:40.586668 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 14:35:40.586932 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260810 14:35:40.589078 25534 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35263
I20260810 14:35:40.589344 25534 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:40.589902 25534 heartbeater.cc:511] Master 127.21.43.126:35263 requested a full tablet report, sending...
I20260810 14:35:40.591022 25035 ts_manager.cc:194] Registered new tserver with Master: c2f575df909243b3a3d5597ecf113108 (127.21.43.67:41283)
I20260810 14:35:40.591842 25035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:58921
I20260810 14:35:40.592549 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[24986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372540, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:40.622344 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:40.613171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.21.43.126:35263)
0810 14:35:40.613316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:40.613320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:40.613564 (+ 244us) server_negotiation.cc:407] Connection header received
0810 14:35:40.613647 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:40.613650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:40.613701 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:40.613778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:40.614483 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.615026 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.615957 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:40.616184 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:40.616431 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:40.619124 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:40.619127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:40.619129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:40.619216 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:40.620641 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.621078 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.621080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.621081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.621157 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:40.621437 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:40.621439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:40.621440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:40.621667 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:40.621781 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:40.622073 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 14:35:40.622212 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 14:35:40.622771 25035 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36166
I20260810 14:35:40.623737 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25104
I20260810 14:35:40.631863 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25243
I20260810 14:35:40.643674 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25391
I20260810 14:35:40.651841 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25000
2026-08-10T14:35:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2310 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[25548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[25548](info): set up 2 sockets
Aug 10 14:35:40 dist-test-slave-chsk krb5kdc[25548](info): commencing operation
krb5kdc: starting...
W20260810 14:35:41.766907 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:41 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:41.915472 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39175
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:42.032260 25563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:42.032604 25563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:42.032691 25563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:42.036907 25563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:42.037012 25563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:42.037048 25563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:42.037107 25563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:42.037161 25563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:42.044169 25563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:42.045369 25563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:42.046478 25563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:42.055405 25568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.055394 25569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.055487 25563 server_base.cc:1034] running on GCE node
W20260810 14:35:42.055394 25571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.056313 25563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:42.057524 25563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:42.058756 25563 hybrid_clock.cc:648] HybridClock initialized: now 1786372542058718 us; error 61 us; skew 500 ppm
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:42.061897 25563 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:42.062965 25563 webserver.cc:460] Webserver started at http://127.21.43.126:46497/ using document root <none> and password file <none>
I20260810 14:35:42.063632 25563 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:42.063732 25563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:42.063979 25563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:42.066185 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "962eeb59febb41bc822801c81c2d416a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "4ac4f44bf631a09de3cdb2560066f394"
server_key_iv: "fa5405f39745942ac10b151e31dd5638"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.066792 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "962eeb59febb41bc822801c81c2d416a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "4ac4f44bf631a09de3cdb2560066f394"
server_key_iv: "fa5405f39745942ac10b151e31dd5638"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.070719 25563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 14:35:42.073469 25581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.074822 25563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:42.075011 25563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "962eeb59febb41bc822801c81c2d416a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "4ac4f44bf631a09de3cdb2560066f394"
server_key_iv: "fa5405f39745942ac10b151e31dd5638"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.075181 25563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:42.105216 25563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:42.109040 25563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:42.109568 25563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:42.118412 25563 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:39175
I20260810 14:35:42.118432 25641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:39175 every 8 connection(s)
I20260810 14:35:42.119508 25563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:42.123383 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:42.124193 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25563
I20260810 14:35:42.124342 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:42.124636 21677 external_mini_cluster.cc:1428] Setting key 60eede61dc1b8ab7c9e7987c2a4cd9be
I20260810 14:35:42.130697 25642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a: Bootstrap starting.
I20260810 14:35:42.133935 25642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a: Neither blocks nor log segments found. Creating new log.
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:42.134954 25642 log.cc:824] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:42.137352 25642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a: No bootstrap required, opened a new log
I20260810 14:35:42.140249 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.126147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.21.43.126:39175)
0810 14:35:42.126610 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.126626 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.126666 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:35:42.127436 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.127456 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.127804 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.128092 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.129136 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.130175 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.131452 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.131959 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.132126 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.135219 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.135225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.135240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.135277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.137188 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.137779 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.137785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.137789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.137902 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.138231 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.138236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.138238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.138662 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.138891 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.139191 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.139495 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":144,"threads_started":1}
I20260810 14:35:42.141399 25642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } }
I20260810 14:35:42.141707 25642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:42.141760 25642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962eeb59febb41bc822801c81c2d416a, State: Initialized, Role: FOLLOWER
I20260810 14:35:42.142277 25642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } }
I20260810 14:35:42.142444 25642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:42.142546 25642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:42.142668 25642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:42.143766 25642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } }
I20260810 14:35:42.144141 25642 leader_election.cc:302] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 962eeb59febb41bc822801c81c2d416a; no voters:
I20260810 14:35:42.144558 25642 leader_election.cc:288] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:42.144719 25647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:42.144943 25647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [term 1 LEADER]: Becoming Leader. State: Replica: 962eeb59febb41bc822801c81c2d416a, State: Running, Role: LEADER
I20260810 14:35:42.145255 25647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } }
I20260810 14:35:42.145876 25642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:42.147277 25648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "962eeb59febb41bc822801c81c2d416a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } } }
I20260810 14:35:42.147414 25648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:42.147830 25657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:42.148247 25649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962eeb59febb41bc822801c81c2d416a. Latest consensus state: current_term: 1 leader_uuid: "962eeb59febb41bc822801c81c2d416a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "962eeb59febb41bc822801c81c2d416a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39175 } } }
I20260810 14:35:42.148347 25649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:42.151818 25657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:42.158179 25657 catalog_manager.cc:1347] Generated new cluster ID: d07467722e2747628a48b61475bbecba
I20260810 14:35:42.158288 25657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:42.170042 25657 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:42.171749 25657 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:42.181938 25657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962eeb59febb41bc822801c81c2d416a: Generated new TSK 0
I20260810 14:35:42.182751 25657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:42.218389 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:42.343077 25669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:42.343400 25669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:42.343447 25669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:42.347995 25669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:42.348105 25669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:42.348225 25669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:42.353956 25669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:42.355329 25669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:42.356455 25669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:42.366837 25674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.366853 25677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.367193 25669 server_base.cc:1034] running on GCE node
W20260810 14:35:42.366833 25675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.367635 25669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:42.368330 25669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:42.369540 25669 hybrid_clock.cc:648] HybridClock initialized: now 1786372542369522 us; error 33 us; skew 500 ppm
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:42.372689 25669 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:42.373656 25669 webserver.cc:460] Webserver started at http://127.21.43.65:37935/ using document root <none> and password file <none>
I20260810 14:35:42.374260 25669 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:42.374358 25669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:42.374599 25669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:42.376766 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "2cf976b93bb649c797b113cec03cfd61"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "76c0dc70a8abd4ee765829a1d256bc7b"
server_key_iv: "2907bdb8aa97cb69db129c6e176def01"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.377372 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "2cf976b93bb649c797b113cec03cfd61"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "76c0dc70a8abd4ee765829a1d256bc7b"
server_key_iv: "2907bdb8aa97cb69db129c6e176def01"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.381814 25669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:42.384668 25684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.386019 25669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:42.386178 25669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "2cf976b93bb649c797b113cec03cfd61"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "76c0dc70a8abd4ee765829a1d256bc7b"
server_key_iv: "2907bdb8aa97cb69db129c6e176def01"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.386324 25669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:42.398522 25669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:42.402077 25669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:42.402354 25669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:42.403293 25669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:42.404248 25669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:42.404331 25669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.404412 25669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:42.404458 25669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.415004 25669 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39309
I20260810 14:35:42.415023 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39309 every 8 connection(s)
I20260810 14:35:42.416210 25669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:42.416920 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25669
I20260810 14:35:42.417050 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:42.417372 21677 external_mini_cluster.cc:1428] Setting key 5ceaf65a8281fec45c72038bf87c9651
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:42.433501 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.418346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:35041 (local address 127.21.43.126:39175)
0810 14:35:42.418498 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.418503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.419662 (+ 1159us) server_negotiation.cc:407] Connection header received
0810 14:35:42.421237 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.421240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.421300 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.421387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.423283 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.423828 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.425882 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.426137 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.426228 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.429509 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.429512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.429515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.429541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.431084 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.431782 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.431788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.431790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.431875 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.432258 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.432261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.432263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.432468 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.432585 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.433160 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.433313 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:42.434386 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.418862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39175 (local address 127.21.43.65:35041)
0810 14:35:42.419447 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:42.419519 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:42.420932 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:42.421646 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:42.421657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:42.422288 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:42.423051 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.423078 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.423996 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.424003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.425711 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.425738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.426358 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.426365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.426615 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.427447 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:42.427476 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:42.429273 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:42.431324 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.431331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.431348 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.431675 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.431977 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.431981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.431984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.432145 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.432585 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:42.432593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:42.432899 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 14:35:42.433103 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 14:35:42.433506 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260810 14:35:42.435844 25812 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39175
I20260810 14:35:42.436137 25812 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:42.436800 25812 heartbeater.cc:511] Master 127.21.43.126:39175 requested a full tablet report, sending...
I20260810 14:35:42.438874 25599 ts_manager.cc:194] Registered new tserver with Master: 2cf976b93bb649c797b113cec03cfd61 (127.21.43.65:39309)
I20260810 14:35:42.440116 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:35041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:42.451650 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:42.569871 25817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:42.570211 25817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:42.570292 25817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:42.574795 25817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:42.574927 25817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:42.575042 25817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:42.580991 25817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:42.582216 25817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:42.583405 25817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:42.594238 25823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.594273 25825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.594238 25822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.594748 25817 server_base.cc:1034] running on GCE node
I20260810 14:35:42.595412 25817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:42.596436 25817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:42.597671 25817 hybrid_clock.cc:648] HybridClock initialized: now 1786372542597646 us; error 35 us; skew 500 ppm
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:42.600827 25817 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:42.601866 25817 webserver.cc:460] Webserver started at http://127.21.43.66:38859/ using document root <none> and password file <none>
I20260810 14:35:42.602504 25817 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:42.602610 25817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:42.602867 25817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:42.605170 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "26b89e0c1c60479498755b405807234a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "182dcf26e5c66c063eac59f17243cb52"
server_key_iv: "690f07ff72383a6359dd2fde488fdea5"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.605926 25817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "26b89e0c1c60479498755b405807234a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "182dcf26e5c66c063eac59f17243cb52"
server_key_iv: "690f07ff72383a6359dd2fde488fdea5"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.610618 25817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:35:42.613641 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.614946 25817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:42.615099 25817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "26b89e0c1c60479498755b405807234a"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "182dcf26e5c66c063eac59f17243cb52"
server_key_iv: "690f07ff72383a6359dd2fde488fdea5"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.615296 25817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:42.649086 25817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:42.652814 25817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:42.653051 25817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:42.653705 25817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:42.654608 25817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:42.654685 25817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.654762 25817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:42.654820 25817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.664414 25817 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45345
I20260810 14:35:42.664435 25960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45345 every 8 connection(s)
I20260810 14:35:42.665396 25817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:42.670286 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25817
I20260810 14:35:42.670424 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:42.670691 21677 external_mini_cluster.cc:1428] Setting key 3207e50ccfec462c148673db5869e178
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:35:42.684196 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.667335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:56313 (local address 127.21.43.126:39175)
0810 14:35:42.667490 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.667494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.668373 (+ 879us) server_negotiation.cc:407] Connection header received
0810 14:35:42.670406 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.670410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.670474 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.670586 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.672039 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.672627 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.674472 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.674903 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.675037 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.679343 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.679350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.679355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.679398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.681352 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.682153 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.682157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.682159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.682267 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.682729 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.682733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.682735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.682952 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.683036 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.683849 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.684009 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:42.685135 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.667745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39175 (local address 127.21.43.66:56313)
0810 14:35:42.668212 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:42.668257 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:42.670178 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:42.670796 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:42.670805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:42.671297 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:42.671849 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.671895 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.672774 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.672780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.674299 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.674315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.675127 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.675131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.675522 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.676319 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:42.676336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:42.678982 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:42.681549 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.681556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.681596 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.681978 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.682409 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.682414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.682419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.682597 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.683087 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:42.683093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:42.683457 (+ 364us) client_negotiation.cc:769] Sending connection context
0810 14:35:42.683774 (+ 317us) client_negotiation.cc:240] Negotiation successful
0810 14:35:42.684147 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260810 14:35:42.686563 25961 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39175
I20260810 14:35:42.686861 25961 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:42.687480 25961 heartbeater.cc:511] Master 127.21.43.126:39175 requested a full tablet report, sending...
I20260810 14:35:42.688758 25599 ts_manager.cc:194] Registered new tserver with Master: 26b89e0c1c60479498755b405807234a (127.21.43.66:45345)
I20260810 14:35:42.689443 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:56313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:42.706847 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:42.827230 25967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:42.827553 25967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:42.827631 25967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:42.831818 25967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:42.831907 25967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:42.832000 25967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:42.837476 25967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:42.838687 25967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:42.839847 25967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:42.850240 25975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.850235 25972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:42.850238 25973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:42.850692 25967 server_base.cc:1034] running on GCE node
I20260810 14:35:42.851135 25967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:42.851943 25967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:42.853132 25967 hybrid_clock.cc:648] HybridClock initialized: now 1786372542853119 us; error 25 us; skew 500 ppm
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:42.856033 25967 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:42.856986 25967 webserver.cc:460] Webserver started at http://127.21.43.67:40883/ using document root <none> and password file <none>
I20260810 14:35:42.857544 25967 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:42.857606 25967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:42.857805 25967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:42.859977 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "f02660c15e524463b88ab7f5b4800419"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "dbac8236d68240b701a64870f0c32b4b"
server_key_iv: "8f9b0abe1dc555e7f755bed4e6e6cbed"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.860522 25967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "f02660c15e524463b88ab7f5b4800419"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "dbac8236d68240b701a64870f0c32b4b"
server_key_iv: "8f9b0abe1dc555e7f755bed4e6e6cbed"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.864531 25967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:42.866968 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.868319 25967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:35:42.868463 25967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "f02660c15e524463b88ab7f5b4800419"
format_stamp: "Formatted at 2026-08-10 14:35:42 on dist-test-slave-chsk"
server_key: "dbac8236d68240b701a64870f0c32b4b"
server_key_iv: "8f9b0abe1dc555e7f755bed4e6e6cbed"
server_key_version: "encryptionkey@0"
I20260810 14:35:42.868578 25967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:42.886612 25967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:42.890591 25967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:42.890836 25967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:42.891613 25967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:42.892526 25967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:42.892575 25967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.892647 25967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:42.892686 25967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:42.903096 25967 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:46463
I20260810 14:35:42.903119 26107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:46463 every 8 connection(s)
I20260810 14:35:42.904242 25967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:42.905424 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 25967
I20260810 14:35:42.905571 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:42.905884 21677 external_mini_cluster.cc:1428] Setting key f186a81cfca86a9d2b8c625adae90161
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:42.920822 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.906719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:50325 (local address 127.21.43.126:39175)
0810 14:35:42.906895 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.906900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.908189 (+ 1289us) server_negotiation.cc:407] Connection header received
0810 14:35:42.909665 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.909668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.909724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.909800 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.911490 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.912039 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.913274 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.913510 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.913633 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.916440 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.916444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.916446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.916480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.918032 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.918692 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.918695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.918696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.918847 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.919355 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.919361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.919365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.919673 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.919787 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.920517 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.920655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:35:42.921761 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.907436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39175 (local address 127.21.43.67:50325)
0810 14:35:42.908037 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:42.908099 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:42.909435 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:42.909993 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:42.910000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:42.910590 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:42.911336 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.911360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.912190 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.912193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.913127 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:42.913137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.913693 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.913697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:42.913892 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.914553 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:42.914571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:42.916208 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:42.918174 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.918181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.918199 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.918494 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.918952 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:42.918955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:42.918957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:42.919218 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:42.919804 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:42.919810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:42.920254 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 14:35:42.920458 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 14:35:42.920863 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":139,"threads_started":1}
I20260810 14:35:42.923005 26108 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39175
I20260810 14:35:42.923309 26108 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:42.923803 26108 heartbeater.cc:511] Master 127.21.43.126:39175 requested a full tablet report, sending...
I20260810 14:35:42.924877 25599 ts_manager.cc:194] Registered new tserver with Master: f02660c15e524463b88ab7f5b4800419 (127.21.43.67:46463)
I20260810 14:35:42.925427 25599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:50325
I20260810 14:35:42.932802 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:42.961918 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.951751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.21.43.65:39309)
0810 14:35:42.952038 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.952047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.952079 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:35:42.952143 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.952149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.952341 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.952465 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.953138 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.953893 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.954757 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.955124 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.955274 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.957795 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.957805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.957819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.957847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.959904 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.960422 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.960427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.960429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.960522 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.960864 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.960867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.960868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.961150 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.961332 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.961627 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.961771 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":109,"threads_started":1}
Aug 10 14:35:42 dist-test-slave-chsk krb5kdc[25548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:42.974141 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:42.965291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.21.43.126:39175)
0810 14:35:42.965432 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:42.965436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:42.965451 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:35:42.965549 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:42.965551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:42.965598 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:42.965664 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:42.966295 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.966825 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.967725 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:42.967974 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:42.968080 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:42.970742 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:42.970745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:42.970748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:42.970778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:42.972477 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.972889 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.972892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.972893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.972974 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:42.973267 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:42.973271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:42.973273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:42.973539 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:42.973663 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:42.973896 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 14:35:42.974006 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 14:35:42.974788 25599 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43068
I20260810 14:35:42.975478 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25669
I20260810 14:35:42.984719 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25817
I20260810 14:35:42.995864 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25967
I20260810 14:35:43.005213 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 25563
2026-08-10T14:35:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2353 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:43 dist-test-slave-chsk krb5kdc[26122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:43 dist-test-slave-chsk krb5kdc[26122](info): set up 2 sockets
Aug 10 14:35:43 dist-test-slave-chsk krb5kdc[26122](info): commencing operation
krb5kdc: starting...
W20260810 14:35:44.122359 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:44.258422 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37171
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:37171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:44.372792 26135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:44.373123 26135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:44.373212 26135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:44.377238 26135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:44.377337 26135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:44.377377 26135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:44.377445 26135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:44.377485 26135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:44.382848 26135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:37171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:44.383941 26135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:44.384994 26135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:44.393719 26142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:44.393709 26144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:44.393709 26140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:44.393991 26135 server_base.cc:1034] running on GCE node
I20260810 14:35:44.394740 26135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:44.396142 26135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:44.397351 26135 hybrid_clock.cc:648] HybridClock initialized: now 1786372544397325 us; error 49 us; skew 500 ppm
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:44.400283 26135 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:44.401101 26135 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:44.401211 26135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:44.401469 26135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:44.403800 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "13b123dd08194747800b096fe5e32158"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "18bbdbf64b4b4e12b2652570ad0e26f3"
server_key_iv: "2f8a57bf52f00ce77676c81be24df1b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.404423 26135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "13b123dd08194747800b096fe5e32158"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "18bbdbf64b4b4e12b2652570ad0e26f3"
server_key_iv: "2f8a57bf52f00ce77676c81be24df1b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.408420 26135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:44.410964 26150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.412338 26135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:35:44.412514 26135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "13b123dd08194747800b096fe5e32158"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "18bbdbf64b4b4e12b2652570ad0e26f3"
server_key_iv: "2f8a57bf52f00ce77676c81be24df1b8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.412648 26135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:44.432807 26135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:44.436312 26135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:44.436558 26135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:44.444480 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:37171 every 8 connection(s)
I20260810 14:35:44.444484 26135 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:37171
I20260810 14:35:44.445250 26135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:44.445645 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26135
I20260810 14:35:44.445798 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:44.446102 21677 external_mini_cluster.cc:1428] Setting key 3291f1dc61616438984f0f5a87240cd9
I20260810 14:35:44.448487 26212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:44.454510 26212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: Bootstrap starting.
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:44.457149 26212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:44.457907 26212 log.cc:824] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:44.460371 26212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: No bootstrap required, opened a new log
I20260810 14:35:44.461374 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:44.447728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.21.43.126:37171)
0810 14:35:44.448211 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:44.448226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:44.448262 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:35:44.448980 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:44.448999 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:44.449334 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:44.449688 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:44.450556 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.451633 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.452524 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.453019 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.453198 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:44.456196 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:44.456202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:44.456214 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:44.456243 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:44.458090 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.458641 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.458645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.458649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.458792 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.459098 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.459100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.459102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.459640 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:44.459840 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:44.460105 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:35:44.460419 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":339,"thread_start_us":159,"threads_started":1}
I20260810 14:35:44.464629 26212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } }
I20260810 14:35:44.464852 26212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:44.464887 26212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b123dd08194747800b096fe5e32158, State: Initialized, Role: FOLLOWER
I20260810 14:35:44.465245 26212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } }
I20260810 14:35:44.465353 26212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:44.465400 26212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:44.465471 26212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:44.466344 26212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } }
I20260810 14:35:44.466634 26212 leader_election.cc:302] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b123dd08194747800b096fe5e32158; no voters:
I20260810 14:35:44.466912 26212 leader_election.cc:288] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:44.467059 26217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:44.467352 26217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [term 1 LEADER]: Becoming Leader. State: Replica: 13b123dd08194747800b096fe5e32158, State: Running, Role: LEADER
I20260810 14:35:44.467643 26217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } }
I20260810 14:35:44.468250 26212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:44.469445 26219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b123dd08194747800b096fe5e32158. Latest consensus state: current_term: 1 leader_uuid: "13b123dd08194747800b096fe5e32158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } } }
I20260810 14:35:44.469578 26219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:44.469961 26228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:44.469907 26218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b123dd08194747800b096fe5e32158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b123dd08194747800b096fe5e32158" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37171 } } }
I20260810 14:35:44.470443 26218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:44.472820 26228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:44.479086 26228 catalog_manager.cc:1347] Generated new cluster ID: c934e4dc0d194322a416a3f521037c29
I20260810 14:35:44.479198 26228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:44.485663 26228 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:44.486591 26228 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:44.492580 26228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: Generated new TSK 0
I20260810 14:35:44.493288 26228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:44.527890 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:44.643482 26242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:44.643757 26242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:44.643805 26242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:44.647908 26242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:44.647964 26242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:44.648052 26242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:44.653401 26242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:44.654621 26242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:44.655719 26242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:44.665467 26248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:44.665514 26250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:44.665469 26247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:44.665704 26242 server_base.cc:1034] running on GCE node
I20260810 14:35:44.666249 26242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:44.666937 26242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:44.668129 26242 hybrid_clock.cc:648] HybridClock initialized: now 1786372544668108 us; error 41 us; skew 500 ppm
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:44.671291 26242 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:44.672075 26242 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:44.672178 26242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:44.672426 26242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:44.674549 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "85231d99422a47f6a28a7b4d5dd22f57"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "be2e7346cfdbd6d4b1fc4e1bcdacafb8"
server_key_iv: "f4ce7f9e8d9d6c7e0e397c49940ab1a8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.675199 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "85231d99422a47f6a28a7b4d5dd22f57"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "be2e7346cfdbd6d4b1fc4e1bcdacafb8"
server_key_iv: "f4ce7f9e8d9d6c7e0e397c49940ab1a8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.679111 26242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:44.681747 26256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.683091 26242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:44.683279 26242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "85231d99422a47f6a28a7b4d5dd22f57"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "be2e7346cfdbd6d4b1fc4e1bcdacafb8"
server_key_iv: "f4ce7f9e8d9d6c7e0e397c49940ab1a8"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.683408 26242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:44.702891 26242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:44.706305 26242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:44.706539 26242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:44.707248 26242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:44.708127 26242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:44.708204 26242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.708277 26242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:44.708323 26242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.718039 26242 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:36699
I20260810 14:35:44.718055 26382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:36699 every 8 connection(s)
I20260810 14:35:44.718678 26242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:44.725668 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26242
I20260810 14:35:44.725812 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:44.726101 21677 external_mini_cluster.cc:1428] Setting key 9404596ce5f1fcfe9bd66431e7868592
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:44.739557 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:44.720568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:60469 (local address 127.21.43.126:37171)
0810 14:35:44.720731 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:44.720734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:44.721510 (+ 776us) server_negotiation.cc:407] Connection header received
0810 14:35:44.722895 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:44.722899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:44.722961 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:44.723074 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:44.724300 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.724853 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.726407 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.726739 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.726854 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:44.735303 (+ 8449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:44.735307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:44.735309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:44.735338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:44.736935 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.737744 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.737747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.737748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.737836 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.738252 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.738255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.738256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.738463 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:44.738557 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:44.739271 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 14:35:44.739399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:44.740659 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:44.720862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37171 (local address 127.21.43.65:60469)
0810 14:35:44.721357 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:44.721400 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:44.722673 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:44.723281 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:44.723288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:44.723658 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:44.724143 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:44.724158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.724992 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.725000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:44.726266 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:44.726278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.726922 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.726926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:44.727110 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:44.732873 (+ 5763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:44.732905 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:44.735018 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:44.737127 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:44.737135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:44.737157 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:44.737600 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:44.737937 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:44.737940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:44.737941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:44.738161 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:44.738610 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:44.738616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:44.738912 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 14:35:44.739124 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 14:35:44.739490 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260810 14:35:44.742205 26383 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37171
I20260810 14:35:44.742483 26383 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:44.742997 26383 heartbeater.cc:511] Master 127.21.43.126:37171 requested a full tablet report, sending...
I20260810 14:35:44.744853 26171 ts_manager.cc:194] Registered new tserver with Master: 85231d99422a47f6a28a7b4d5dd22f57 (127.21.43.65:36699)
I20260810 14:35:44.746647 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:60469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:44.758725 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:44.873684 26388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:44.873980 26388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:44.874027 26388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:44.878319 26388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:44.878384 26388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:44.878480 26388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:44.883989 26388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:44.885231 26388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:44.886410 26388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:44.896484 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:44.896539 26388 server_base.cc:1034] running on GCE node
W20260810 14:35:44.896499 26394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:44.896508 26393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:44.897279 26388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:44.897958 26388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:44.899262 26388 hybrid_clock.cc:648] HybridClock initialized: now 1786372544899232 us; error 28 us; skew 500 ppm
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:44.902470 26388 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:44.903385 26388 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:44.903502 26388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:44.903769 26388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:44.906068 26388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "ad1df8ce680144a694c97541b918f4a8"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "348ede57cdfd63511e59cb0e1e0ebf47"
server_key_iv: "60342bd50cd769994714ce1698870a5c"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.906735 26388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "ad1df8ce680144a694c97541b918f4a8"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "348ede57cdfd63511e59cb0e1e0ebf47"
server_key_iv: "60342bd50cd769994714ce1698870a5c"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.910946 26388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:35:44.913728 26404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.915066 26388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:44.915261 26388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "ad1df8ce680144a694c97541b918f4a8"
format_stamp: "Formatted at 2026-08-10 14:35:44 on dist-test-slave-chsk"
server_key: "348ede57cdfd63511e59cb0e1e0ebf47"
server_key_iv: "60342bd50cd769994714ce1698870a5c"
server_key_version: "encryptionkey@0"
I20260810 14:35:44.915405 26388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:44.929361 26388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:44.932909 26388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:44.933171 26388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:44.933812 26388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:44.934715 26388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:44.934803 26388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.934878 26388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:44.934937 26388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:44.944430 26388 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45057
I20260810 14:35:44.944447 26531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45057 every 8 connection(s)
I20260810 14:35:44.945078 26388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:44.946496 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26388
I20260810 14:35:44.946635 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:44.946969 21677 external_mini_cluster.cc:1428] Setting key 1ea4f47de7d7497b3473e1243424956d
Aug 10 14:35:44 dist-test-slave-chsk krb5kdc[26122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:35:44.960695 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:44.946964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:39433 (local address 127.21.43.126:37171)
0810 14:35:44.947119 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:44.947180 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:44.948199 (+ 1019us) server_negotiation.cc:407] Connection header received
0810 14:35:44.949503 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:44.949509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:44.949571 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:44.949665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:44.951033 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.951644 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.953439 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.953707 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.953813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:44.956881 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:44.956888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:44.956891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:44.956923 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:44.958556 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.959046 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.959049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.959050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.959194 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:44.959574 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:44.959577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:44.959579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:44.959780 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:44.959897 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:44.960439 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 14:35:44.960567 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:44.961977 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:44.947496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37171 (local address 127.21.43.66:39433)
0810 14:35:44.948011 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:44.948075 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:44.949279 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:44.949821 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:44.949827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:44.950203 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:44.950864 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:44.950887 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.951782 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.951787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:44.953302 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:44.953317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:44.953897 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:44.953905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:44.954139 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:44.954764 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:44.954790 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:44.956696 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:44.958685 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:44.958690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:44.958702 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:44.958952 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:44.959291 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:44.959293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:44.959294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:44.959462 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:44.959896 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:44.959903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:44.960160 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 14:35:44.960436 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 14:35:44.960929 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":152,"threads_started":1}
I20260810 14:35:44.963323 26532 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37171
I20260810 14:35:44.963603 26532 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:44.964201 26532 heartbeater.cc:511] Master 127.21.43.126:37171 requested a full tablet report, sending...
I20260810 14:35:44.965447 26171 ts_manager.cc:194] Registered new tserver with Master: ad1df8ce680144a694c97541b918f4a8 (127.21.43.66:45057)
I20260810 14:35:44.966061 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:39433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:44.978669 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:45.098299 26537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:45.098599 26537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:45.098649 26537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:45.102880 26537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:45.102964 26537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:45.103029 26537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:45.108807 26537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:45.109952 26537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:45.111009 26537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:45.121234 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:45.121235 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:45.121234 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:45.121310 26537 server_base.cc:1034] running on GCE node
I20260810 14:35:45.122009 26537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:45.122702 26537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:45.123898 26537 hybrid_clock.cc:648] HybridClock initialized: now 1786372545123875 us; error 36 us; skew 500 ppm
Aug 10 14:35:45 dist-test-slave-chsk krb5kdc[26122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:45.126879 26537 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:45.127709 26537 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:45.127816 26537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:45.128067 26537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:45.130158 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "14c4ecc499154479970d255a0daf4af9"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "eb4fa187765abade8908841ad9a702d4"
server_key_iv: "17bc65d77850e7216b62e760446c9fb6"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.130738 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "14c4ecc499154479970d255a0daf4af9"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "eb4fa187765abade8908841ad9a702d4"
server_key_iv: "17bc65d77850e7216b62e760446c9fb6"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.134593 26537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:35:45.137204 26552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:45.138408 26537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:45.138554 26537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "14c4ecc499154479970d255a0daf4af9"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "eb4fa187765abade8908841ad9a702d4"
server_key_iv: "17bc65d77850e7216b62e760446c9fb6"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.138675 26537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:45.154070 26537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:45.157761 26537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:45.157991 26537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:45.158620 26537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:45.159586 26537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:45.159637 26537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:45.159710 26537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:45.159744 26537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:45.168813 26537 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:42623
I20260810 14:35:45.168848 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:42623 every 8 connection(s)
I20260810 14:35:45.169538 26537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:45.176304 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26537
I20260810 14:35:45.176445 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:45.176754 21677 external_mini_cluster.cc:1428] Setting key c1658bad5c7090f4a322ae30f38d28fe
Aug 10 14:35:45 dist-test-slave-chsk krb5kdc[26122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:45.183836 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.171365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45971 (local address 127.21.43.126:37171)
0810 14:35:45.171517 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.171521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.172350 (+ 829us) server_negotiation.cc:407] Connection header received
0810 14:35:45.173657 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.173660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.173719 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.173825 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:45.174956 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.175554 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.177125 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.177424 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.177518 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.180316 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:45.180319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:45.180321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:45.180347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:45.181766 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:45.182340 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:45.182342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:45.182343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:45.182417 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:45.182729 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:45.182731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:45.182732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:45.182904 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:45.182980 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.183606 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.183713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:45.184684 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.171740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37171 (local address 127.21.43.67:45971)
0810 14:35:45.172201 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.172243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.173437 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.173993 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.174001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.174343 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:45.174806 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.174819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.175685 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.175691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.176954 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.176966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.177620 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.177625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.177850 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.178583 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:45.178603 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:45.180151 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:45.181895 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:45.181901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:45.181914 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:45.182243 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:45.182501 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:45.182503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:45.182504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:45.182646 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:45.183024 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:45.183030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:45.183363 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.183546 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.183819 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260810 14:35:45.185906 26679 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37171
I20260810 14:35:45.186167 26679 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:45.186725 26679 heartbeater.cc:511] Master 127.21.43.126:37171 requested a full tablet report, sending...
I20260810 14:35:45.187937 26171 ts_manager.cc:194] Registered new tserver with Master: 14c4ecc499154479970d255a0daf4af9 (127.21.43.67:42623)
I20260810 14:35:45.188482 26171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:45971
I20260810 14:35:45.191108 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:45.200453 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.193180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.21.43.126:37171)
0810 14:35:45.193364 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.193368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.193446 (+ 78us) server_negotiation.cc:407] Connection header received
0810 14:35:45.193550 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.193552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.193588 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.193662 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:45.194358 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.194859 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.195737 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.195940 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.196017 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.197310 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:45.197313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:45.197315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:45.197338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:45.198788 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:45.199262 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:45.199265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:45.199266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:45.199339 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:45.199592 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:45.199594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:45.199596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:45.199812 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:45.199914 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.200147 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.200240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 14:35:45.203390 26171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:45.206419 26171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:45.221562 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.215310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:42623 (local address 127.0.0.1:33116)
0810 14:35:45.215706 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.215761 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.215894 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.216499 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.216506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.216536 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:45.216706 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.216716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.218411 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.218414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.219815 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.219826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.220966 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.220970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.221089 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.221140 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.221251 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.221389 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260810 14:35:45.221732 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.215417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.21.43.67:42623)
0810 14:35:45.215849 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.215861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.215885 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:35:45.216050 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.216057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.216304 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.216877 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:45.217104 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.218258 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.219963 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.220833 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.220940 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.221105 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.221472 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.221575 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":188,"threads_started":1}
I20260810 14:35:45.221760 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.216510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36699 (local address 127.0.0.1:37098)
0810 14:35:45.216855 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.216873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.216965 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.217579 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.217583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.217602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:45.217708 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.217719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.219021 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.219025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.220305 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.220317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.221252 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.221256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.221346 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.221373 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.221452 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.221527 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":137,"thread_start_us":94,"threads_started":1}
I20260810 14:35:45.222244 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.216177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45057 (local address 127.0.0.1:48638)
0810 14:35:45.216816 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.216831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.216904 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.217767 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.217772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.217861 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:45.217926 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.217931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.219331 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.219334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.220569 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.220589 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.221803 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.221807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.221919 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.221947 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.222038 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.222119 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260810 14:35:45.222771 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.216929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.21.43.65:36699)
0810 14:35:45.217208 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.217213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.217233 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:35:45.217290 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.217294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.217430 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.217595 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:45.217840 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.218872 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.220476 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.221105 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.221245 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.221379 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.222501 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.222615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260810 14:35:45.222923 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.216398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.21.43.66:45057)
0810 14:35:45.217215 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.217226 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.217247 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:35:45.217325 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.217329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.217506 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.217649 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:45.218050 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.219121 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.220734 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.221547 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.222450 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.222614 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.222663 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.222750 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":131,"threads_started":1}
I20260810 14:35:45.225308 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 62bf6acb6f324797855278d8103a90da (DEFAULT_TABLE table=test-table [id=8acc52623cfa4a2b83dd545671ec0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:45.225344 26605 tablet_service.cc:1467] Processing CreateTablet for tablet 62bf6acb6f324797855278d8103a90da (DEFAULT_TABLE table=test-table [id=8acc52623cfa4a2b83dd545671ec0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:45.225344 26308 tablet_service.cc:1467] Processing CreateTablet for tablet 62bf6acb6f324797855278d8103a90da (DEFAULT_TABLE table=test-table [id=8acc52623cfa4a2b83dd545671ec0840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:45.226562 26605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bf6acb6f324797855278d8103a90da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:45.226559 26308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bf6acb6f324797855278d8103a90da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:45.226559 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bf6acb6f324797855278d8103a90da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:45.233948 26696 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Bootstrap starting.
I20260810 14:35:45.235666 26698 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Bootstrap starting.
I20260810 14:35:45.236927 26696 tablet_bootstrap.cc:654] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:45.237473 26697 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Bootstrap starting.
I20260810 14:35:45.237969 26698 tablet_bootstrap.cc:654] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:45.238961 26696 log.cc:824] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:45.239262 26698 log.cc:824] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:45.239833 26697 tablet_bootstrap.cc:654] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:45.240913 26697 log.cc:824] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:45.241326 26696 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: No bootstrap required, opened a new log
I20260810 14:35:45.241325 26698 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: No bootstrap required, opened a new log
I20260810 14:35:45.241537 26698 ts_tablet_manager.cc:1397] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:45.241537 26696 ts_tablet_manager.cc:1397] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260810 14:35:45.242923 26697 tablet_bootstrap.cc:492] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: No bootstrap required, opened a new log
I20260810 14:35:45.243187 26697 ts_tablet_manager.cc:1397] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:35:45.245657 26696 raft_consensus.cc:348] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.245664 26698 raft_consensus.cc:348] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.245956 26698 raft_consensus.cc:374] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:45.245956 26696 raft_consensus.cc:374] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:45.246018 26696 raft_consensus.cc:729] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1df8ce680144a694c97541b918f4a8, State: Initialized, Role: FOLLOWER
I20260810 14:35:45.246018 26698 raft_consensus.cc:729] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85231d99422a47f6a28a7b4d5dd22f57, State: Initialized, Role: FOLLOWER
I20260810 14:35:45.246511 26698 consensus_queue.cc:260] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.246511 26696 consensus_queue.cc:260] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.246706 26697 raft_consensus.cc:348] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.246981 26697 raft_consensus.cc:374] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:45.247057 26697 raft_consensus.cc:729] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4ecc499154479970d255a0daf4af9, State: Initialized, Role: FOLLOWER
I20260810 14:35:45.247388 26383 heartbeater.cc:503] Master 127.21.43.126:37171 was elected leader, sending a full tablet report...
I20260810 14:35:45.247593 26696 ts_tablet_manager.cc:1428] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260810 14:35:45.247535 26697 consensus_queue.cc:260] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.247965 26532 heartbeater.cc:503] Master 127.21.43.126:37171 was elected leader, sending a full tablet report...
I20260810 14:35:45.248518 26697 ts_tablet_manager.cc:1428] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 14:35:45.248657 26698 ts_tablet_manager.cc:1428] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 14:35:45.248817 26679 heartbeater.cc:503] Master 127.21.43.126:37171 was elected leader, sending a full tablet report...
W20260810 14:35:45.420562 26680 tablet.cc:2369] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:45.438903 26704 raft_consensus.cc:482] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:45.439203 26704 raft_consensus.cc:504] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.440623 26704 leader_election.cc:288] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad1df8ce680144a694c97541b918f4a8 (127.21.43.66:45057), 85231d99422a47f6a28a7b4d5dd22f57 (127.21.43.65:36699)
I20260810 14:35:45.445209 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.441018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45057 (local address 127.21.43.67:34891)
0810 14:35:45.441190 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.441210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.441354 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.441722 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.441729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.441753 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:45.441908 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.441916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.442862 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.442865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.443890 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.443899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.444735 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.444739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.444849 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.444871 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.444982 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.445063 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:35:45.445209 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.441142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:34891 (local address 127.21.43.66:45057)
0810 14:35:45.441312 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.441317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.441336 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:35:45.441512 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.441515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.441598 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.441685 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:45.442065 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.442754 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.444036 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.444599 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.444742 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.444802 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.445000 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.445072 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:35:45.445729 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.441518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45051 (local address 127.21.43.65:36699)
0810 14:35:45.441673 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.441679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.441842 (+ 163us) server_negotiation.cc:407] Connection header received
0810 14:35:45.441968 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.441973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.442075 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.442144 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:45.442436 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.443184 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.444446 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.445008 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.445144 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.445192 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.445515 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.445607 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:35:45.446105 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.441339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36699 (local address 127.21.43.67:45051)
0810 14:35:45.441740 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.441755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:45.441854 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:45.442170 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:45.442173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:45.442192 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:45.442313 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.442319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.443303 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.443306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.444258 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:45.444269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.445271 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.445275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:45.445364 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.445388 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:35:45.445883 (+ 495us) client_negotiation.cc:240] Negotiation successful
0810 14:35:45.445960 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":146,"threads_started":1}
I20260810 14:35:45.446043 26481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bf6acb6f324797855278d8103a90da" candidate_uuid: "14c4ecc499154479970d255a0daf4af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1df8ce680144a694c97541b918f4a8" is_pre_election: true
W20260810 14:35:45.446179 26533 tablet.cc:2369] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:45.446400 26481 raft_consensus.cc:2454] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4ecc499154479970d255a0daf4af9 in term 0.
I20260810 14:35:45.446969 26554 leader_election.cc:302] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4ecc499154479970d255a0daf4af9, ad1df8ce680144a694c97541b918f4a8; no voters:
I20260810 14:35:45.447268 26330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bf6acb6f324797855278d8103a90da" candidate_uuid: "14c4ecc499154479970d255a0daf4af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85231d99422a47f6a28a7b4d5dd22f57" is_pre_election: true
I20260810 14:35:45.447332 26704 raft_consensus.cc:2781] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:45.447428 26704 raft_consensus.cc:482] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:45.447481 26704 raft_consensus.cc:3037] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:45.447546 26330 raft_consensus.cc:2454] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4ecc499154479970d255a0daf4af9 in term 0.
I20260810 14:35:45.448809 26704 raft_consensus.cc:504] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.449254 26704 leader_election.cc:288] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [CANDIDATE]: Term 1 election: Requested vote from peers ad1df8ce680144a694c97541b918f4a8 (127.21.43.66:45057), 85231d99422a47f6a28a7b4d5dd22f57 (127.21.43.65:36699)
I20260810 14:35:45.449683 26481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bf6acb6f324797855278d8103a90da" candidate_uuid: "14c4ecc499154479970d255a0daf4af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1df8ce680144a694c97541b918f4a8"
I20260810 14:35:45.449713 26330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bf6acb6f324797855278d8103a90da" candidate_uuid: "14c4ecc499154479970d255a0daf4af9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85231d99422a47f6a28a7b4d5dd22f57"
I20260810 14:35:45.449838 26330 raft_consensus.cc:3037] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:45.449838 26481 raft_consensus.cc:3037] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:45.451395 26481 raft_consensus.cc:2454] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4ecc499154479970d255a0daf4af9 in term 1.
I20260810 14:35:45.451395 26330 raft_consensus.cc:2454] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4ecc499154479970d255a0daf4af9 in term 1.
I20260810 14:35:45.451821 26554 leader_election.cc:302] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4ecc499154479970d255a0daf4af9, ad1df8ce680144a694c97541b918f4a8; no voters:
I20260810 14:35:45.452095 26704 raft_consensus.cc:2781] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:45.452443 26704 raft_consensus.cc:686] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 1 LEADER]: Becoming Leader. State: Replica: 14c4ecc499154479970d255a0daf4af9, State: Running, Role: LEADER
I20260810 14:35:45.452890 26704 consensus_queue.cc:237] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } }
I20260810 14:35:45.456871 26171 catalog_manager.cc:5495] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c4ecc499154479970d255a0daf4af9 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "14c4ecc499154479970d255a0daf4af9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c4ecc499154479970d255a0daf4af9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 42623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 } health_report { overall_health: UNKNOWN } } }
W20260810 14:35:45.470049 26384 tablet.cc:2369] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:45.518968 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.515202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.21.43.67:42623)
0810 14:35:45.515402 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.515406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.515419 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:45.515498 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.515501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.515560 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.515688 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:45.515986 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.516512 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.517477 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:45.517697 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:45.517790 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:45.518098 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:45.518557 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:45.518652 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.518720 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.518779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:35:45.525962 26330 raft_consensus.cc:1261] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 1 FOLLOWER]: Refusing update from remote peer 14c4ecc499154479970d255a0daf4af9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:45.525962 26481 raft_consensus.cc:1261] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 1 FOLLOWER]: Refusing update from remote peer 14c4ecc499154479970d255a0daf4af9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:45.526912 26704 consensus_queue.cc:1035] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85231d99422a47f6a28a7b4d5dd22f57" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:45.527252 26707 consensus_queue.cc:1035] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad1df8ce680144a694c97541b918f4a8" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:45.536396 26711 mvcc.cc:205] Tried to move back new op lower bound from 7316981946464608256 to 7316981946178502656. Current Snapshot: MvccSnapshot[applied={T|T < 7316981946464608256}]
I20260810 14:35:45.537369 26712 mvcc.cc:205] Tried to move back new op lower bound from 7316981946464608256 to 7316981946178502656. Current Snapshot: MvccSnapshot[applied={T|T < 7316981946464608256}]
I20260810 14:35:45.545780 26171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:45.546007 26171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:45.548739 26171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13b123dd08194747800b096fe5e32158: Sending DeleteTablet for 3 replicas of tablet 62bf6acb6f324797855278d8103a90da
I20260810 14:35:45.549767 26605 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bf6acb6f324797855278d8103a90da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:45 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:33116
I20260810 14:35:45.549794 26308 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bf6acb6f324797855278d8103a90da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:45 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:37098
I20260810 14:35:45.549764 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bf6acb6f324797855278d8103a90da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:45 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:48638
I20260810 14:35:45.550400 26720 tablet_replica.cc:330] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57: stopping tablet replica
I20260810 14:35:45.550798 26720 raft_consensus.cc:2229] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:45.550936 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26242
I20260810 14:35:45.551210 26720 raft_consensus.cc:2258] T 62bf6acb6f324797855278d8103a90da P 85231d99422a47f6a28a7b4d5dd22f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:45.552253 26722 tablet_replica.cc:330] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: stopping tablet replica
I20260810 14:35:45.552562 26722 raft_consensus.cc:2229] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:45.552971 26722 raft_consensus.cc:2258] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:45.553359 26721 tablet_replica.cc:330] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: stopping tablet replica
I20260810 14:35:45.553666 26721 raft_consensus.cc:2229] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:45.553963 26721 raft_consensus.cc:2258] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:45.555275 26722 ts_tablet_manager.cc:1905] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:45.558297 26722 ts_tablet_manager.cc:1918] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:45.558382 26722 log.cc:1196] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/62bf6acb6f324797855278d8103a90da
I20260810 14:35:45.558629 26722 ts_tablet_manager.cc:1939] T 62bf6acb6f324797855278d8103a90da P 14c4ecc499154479970d255a0daf4af9: Deleting consensus metadata
I20260810 14:35:45.560034 26152 catalog_manager.cc:4841] TS 14c4ecc499154479970d255a0daf4af9 (127.21.43.67:42623): tablet 62bf6acb6f324797855278d8103a90da (table test-table [id=8acc52623cfa4a2b83dd545671ec0840]) successfully deleted
I20260810 14:35:45.560348 26721 ts_tablet_manager.cc:1905] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:45.562633 26721 ts_tablet_manager.cc:1918] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:45.562767 26721 log.cc:1196] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/62bf6acb6f324797855278d8103a90da
I20260810 14:35:45.563120 26721 ts_tablet_manager.cc:1939] T 62bf6acb6f324797855278d8103a90da P ad1df8ce680144a694c97541b918f4a8: Deleting consensus metadata
I20260810 14:35:45.564353 26152 catalog_manager.cc:4841] TS ad1df8ce680144a694c97541b918f4a8 (127.21.43.66:45057): tablet 62bf6acb6f324797855278d8103a90da (table test-table [id=8acc52623cfa4a2b83dd545671ec0840]) successfully deleted
W20260810 14:35:45.564785 26152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:36699 (error 108)
I20260810 14:35:45.565081 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26388
I20260810 14:35:45.575840 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.575425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36699 (local address 127.0.0.1:37104)
0810 14:35:45.575588 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:45.575708 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 14:35:45.576233 26152 catalog_manager.cc:4584] TS 85231d99422a47f6a28a7b4d5dd22f57 (127.21.43.65:36699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62bf6acb6f324797855278d8103a90da: Network error: Client connection negotiation failed: client connection to 127.21.43.65:36699: connect: Connection refused (error 111)
I20260810 14:35:45.581665 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26537
I20260810 14:35:45.589869 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26135
2026-08-10T14:35:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2584 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T14:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:45Z Disabled control of system clock
I20260810 14:35:45.625999 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35605
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:35605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:35:45.749886 26727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:45.750212 26727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:45.750253 26727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:45.754710 26727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:45.754796 26727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:45.754812 26727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:45.754830 26727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:45.754846 26727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:45.760281 26727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:35605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:45.761560 26727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:45.763283 26727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:45.772390 26732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:45.772457 26735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:45.772490 26727 server_base.cc:1034] running on GCE node
W20260810 14:35:45.772408 26733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:45.773200 26727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:45.774420 26727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:45.775589 26727 hybrid_clock.cc:648] HybridClock initialized: now 1786372545775576 us; error 39 us; skew 500 ppm
I20260810 14:35:45.777745 26727 webserver.cc:460] Webserver started at http://127.21.43.126:43211/ using document root <none> and password file <none>
I20260810 14:35:45.778450 26727 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:45.778524 26727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:45.778791 26727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:45.781102 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "e83788a4f9bc4a7897ea1501ddd4cf40"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "2d8c37ce9638c5ded6bd9ec859a30f05"
server_key_iv: "397a0cd4663664903cf1259b1b6788f8"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.781736 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "e83788a4f9bc4a7897ea1501ddd4cf40"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "2d8c37ce9638c5ded6bd9ec859a30f05"
server_key_iv: "397a0cd4663664903cf1259b1b6788f8"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.785934 26727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:45.788951 26743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:45.790472 26727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:45.790625 26727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "e83788a4f9bc4a7897ea1501ddd4cf40"
format_stamp: "Formatted at 2026-08-10 14:35:45 on dist-test-slave-chsk"
server_key: "2d8c37ce9638c5ded6bd9ec859a30f05"
server_key_iv: "397a0cd4663664903cf1259b1b6788f8"
server_key_version: "encryptionkey@0"
I20260810 14:35:45.790825 26727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:45.796465 26727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:45.797302 26727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:45.797554 26727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:45.806118 26727 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:35605
I20260810 14:35:45.806124 26801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:35605 every 8 connection(s)
I20260810 14:35:45.807420 26727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:45.812885 26802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:45.813162 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26727
I20260810 14:35:45.813303 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:45.813617 21677 external_mini_cluster.cc:1428] Setting key 07a61de4bc12eff4fc97b4e27389252f
I20260810 14:35:45.820828 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:45.815350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.21.43.126:35605)
0810 14:35:45.816214 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:45.816236 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:45.816285 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:35:45.817223 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:45.817252 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:45.817786 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:45.818221 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:45.819080 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:45.819085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:45.819108 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:45.819121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:45.819219 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:45.819245 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:45.819410 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:45.819487 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 14:35:45.819760 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":168,"threads_started":1}
I20260810 14:35:45.821961 26802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: Bootstrap starting.
I20260810 14:35:45.824647 26802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:45.825664 26802 log.cc:824] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:45.828132 26802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: No bootstrap required, opened a new log
I20260810 14:35:45.832183 26802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } }
I20260810 14:35:45.832476 26802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:45.832561 26802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83788a4f9bc4a7897ea1501ddd4cf40, State: Initialized, Role: FOLLOWER
I20260810 14:35:45.833263 26802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } }
I20260810 14:35:45.833489 26802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:45.833575 26802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:45.833725 26802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:45.834980 26802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } }
I20260810 14:35:45.835434 26802 leader_election.cc:302] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e83788a4f9bc4a7897ea1501ddd4cf40; no voters:
I20260810 14:35:45.835826 26802 leader_election.cc:288] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:45.835990 26808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:45.836254 26808 raft_consensus.cc:686] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [term 1 LEADER]: Becoming Leader. State: Replica: e83788a4f9bc4a7897ea1501ddd4cf40, State: Running, Role: LEADER
I20260810 14:35:45.836581 26808 consensus_queue.cc:237] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } }
I20260810 14:35:45.837284 26802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:45.838431 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } } }
I20260810 14:35:45.838581 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:45.838992 26817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:45.840057 26810 sys_catalog.cc:455] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e83788a4f9bc4a7897ea1501ddd4cf40. Latest consensus state: current_term: 1 leader_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e83788a4f9bc4a7897ea1501ddd4cf40" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35605 } } }
I20260810 14:35:45.840173 26810 sys_catalog.cc:458] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:45.843093 26817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:45.851346 26817 catalog_manager.cc:1347] Generated new cluster ID: ab0dd793b65c49a28cdd88c072759f81
I20260810 14:35:45.851534 26817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:45.874536 26817 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:45.874930 26817 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:45.902613 26817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: Generated new TSK 0
I20260810 14:35:45.903683 26817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:35:45.918174 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--builtin_ntp_servers=127.21.43.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:35:46.041302 26827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:46.041602 26827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:46.041671 26827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:46.046038 26827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:46.046109 26827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:46.046191 26827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:46.051541 26827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:46.053054 26827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:46.055068 26827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:46.065667 26835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.065667 26832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.065667 26833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:46.065871 26827 server_base.cc:1034] running on GCE node
I20260810 14:35:46.066339 26827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:46.067023 26827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:46.068212 26827 hybrid_clock.cc:648] HybridClock initialized: now 1786372546068176 us; error 47 us; skew 500 ppm
I20260810 14:35:46.070222 26827 webserver.cc:460] Webserver started at http://127.21.43.65:33371/ using document root <none> and password file <none>
I20260810 14:35:46.070839 26827 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:46.070904 26827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:46.071213 26827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:46.073393 26827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "cb8015176c776f1aaf3a380be7b822d9"
server_key_iv: "4a14dcbb9303cbfe5e4c73fa657731d7"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.073992 26827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "cb8015176c776f1aaf3a380be7b822d9"
server_key_iv: "4a14dcbb9303cbfe5e4c73fa657731d7"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.078033 26827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 14:35:46.080837 26842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.082273 26827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:46.082432 26827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "cb8015176c776f1aaf3a380be7b822d9"
server_key_iv: "4a14dcbb9303cbfe5e4c73fa657731d7"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.082562 26827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:46.087466 26827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:46.088102 26827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:46.088304 26827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:46.088869 26827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:46.089810 26827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:46.089880 26827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.089952 26827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:46.089995 26827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.099673 26827 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:33331
I20260810 14:35:46.099735 26965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:33331 every 8 connection(s)
I20260810 14:35:46.100674 26827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:46.107057 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26827
I20260810 14:35:46.107239 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:46.107373 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.102654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:39097 (local address 127.21.43.126:35605)
0810 14:35:46.102837 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.102842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.104032 (+ 1190us) server_negotiation.cc:407] Connection header received
0810 14:35:46.105445 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.105448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.105492 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.105580 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.106764 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.106768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.106770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.106789 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.106817 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.106831 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.106952 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.107125 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.107214 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:35:46.107597 21677 external_mini_cluster.cc:1428] Setting key e1aa3f3d465d453085101221cd9208f3
I20260810 14:35:46.108556 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.103095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35605 (local address 127.21.43.65:39097)
0810 14:35:46.103875 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.103919 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.105197 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.105757 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.105764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.105855 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.106446 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.106447 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.106496 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.106566 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.106952 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.106961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.106967 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.107133 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.107510 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":122,"threads_started":1}
I20260810 14:35:46.109815 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--builtin_ntp_servers=127.21.43.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:35:46.109838 26967 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35605
I20260810 14:35:46.110113 26967 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:46.110556 26967 heartbeater.cc:511] Master 127.21.43.126:35605 requested a full tablet report, sending...
I20260810 14:35:46.112371 26761 ts_manager.cc:194] Registered new tserver with Master: b6f609a4e51a4f4cb4b037aeae2f07d3 (127.21.43.65:33331)
W20260810 14:35:46.230566 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:46.230937 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:46.231026 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:46.235561 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:46.235680 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:46.235788 26971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:46.241670 26971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:46.242924 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:46.244638 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:46.255409 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.255409 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.255401 26980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:46.255784 26971 server_base.cc:1034] running on GCE node
I20260810 14:35:46.256412 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:46.257104 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:46.258280 26971 hybrid_clock.cc:648] HybridClock initialized: now 1786372546258263 us; error 58 us; skew 500 ppm
I20260810 14:35:46.260398 26971 webserver.cc:460] Webserver started at http://127.21.43.66:40991/ using document root <none> and password file <none>
I20260810 14:35:46.261050 26971 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:46.261121 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:46.261384 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:46.263831 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "9d4b617d154b40f59092662529d50e96"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "8bc72c9e23bcf2d26bf415b331311ac6"
server_key_iv: "82242b21f04e552d7195d3ca859bd007"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.264379 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "9d4b617d154b40f59092662529d50e96"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "8bc72c9e23bcf2d26bf415b331311ac6"
server_key_iv: "82242b21f04e552d7195d3ca859bd007"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.268359 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:46.271096 26986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.272447 26971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:46.272593 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "9d4b617d154b40f59092662529d50e96"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "8bc72c9e23bcf2d26bf415b331311ac6"
server_key_iv: "82242b21f04e552d7195d3ca859bd007"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.272730 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:46.277568 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:46.278200 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:46.278407 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:46.279080 26971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:46.280052 26971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:46.280124 26971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.280196 26971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:46.280251 26971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.289791 26971 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:43231
I20260810 14:35:46.289851 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:43231 every 8 connection(s)
I20260810 14:35:46.290798 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:46.297384 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.292710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:60603 (local address 127.21.43.126:35605)
0810 14:35:46.292860 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.292864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.293995 (+ 1131us) server_negotiation.cc:407] Connection header received
0810 14:35:46.295597 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.295604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.295662 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.295737 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.296972 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.296975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.296977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.296989 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.297008 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.297016 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.297115 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.297224 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.297267 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:35:46.298259 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 26971
I20260810 14:35:46.298386 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:46.298496 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.293120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35605 (local address 127.21.43.66:60603)
0810 14:35:46.293819 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.293860 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.295323 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.295920 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.295927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.296045 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.296704 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.296705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.296748 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.296821 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.297097 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.297104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.297109 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.297246 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.297447 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260810 14:35:46.298732 21677 external_mini_cluster.cc:1428] Setting key a1ed06b40996d8f841de3f991b1b30ec
I20260810 14:35:46.299822 27110 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35605
I20260810 14:35:46.300099 27110 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:46.300606 27110 heartbeater.cc:511] Master 127.21.43.126:35605 requested a full tablet report, sending...
I20260810 14:35:46.301192 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--builtin_ntp_servers=127.21.43.84:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:35:46.301879 26761 ts_manager.cc:194] Registered new tserver with Master: 9d4b617d154b40f59092662529d50e96 (127.21.43.66:43231)
W20260810 14:35:46.417834 27113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:46.418133 27113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:46.418227 27113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:46.422581 27113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:46.422672 27113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:46.422734 27113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:46.427892 27113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:46.429124 27113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:46.430657 27113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:46.441262 27123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.441215 27118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.441215 27119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:46.441805 27113 server_base.cc:1034] running on GCE node
I20260810 14:35:46.442312 27113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:46.443059 27113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:46.444252 27113 hybrid_clock.cc:648] HybridClock initialized: now 1786372546444234 us; error 39 us; skew 500 ppm
I20260810 14:35:46.446398 27113 webserver.cc:460] Webserver started at http://127.21.43.67:43957/ using document root <none> and password file <none>
I20260810 14:35:46.447038 27113 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:46.447120 27113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:46.447458 27113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:46.449736 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "4c83cfdc2df7448994d6f440a2330308"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "f0779ce54141e69cee9ae2417252d398"
server_key_iv: "e1d54fd0d354ad8784a403703ed3452b"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.450342 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "4c83cfdc2df7448994d6f440a2330308"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "f0779ce54141e69cee9ae2417252d398"
server_key_iv: "e1d54fd0d354ad8784a403703ed3452b"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.454557 27113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:35:46.457439 27130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.458913 27113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:46.459059 27113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "4c83cfdc2df7448994d6f440a2330308"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "f0779ce54141e69cee9ae2417252d398"
server_key_iv: "e1d54fd0d354ad8784a403703ed3452b"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.459259 27113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:46.464251 27113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:46.464968 27113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:46.465166 27113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:46.465744 27113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:46.466699 27113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:46.466747 27113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.466815 27113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:46.466854 27113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.475967 27113 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:44403
I20260810 14:35:46.476044 27257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:44403 every 8 connection(s)
I20260810 14:35:46.477005 27113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:46.479022 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27113
I20260810 14:35:46.479174 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:46.479467 21677 external_mini_cluster.cc:1428] Setting key da5db6cf6b6bccb6c4b0c86b5878f9b2
I20260810 14:35:46.483742 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.478920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:37843 (local address 127.21.43.126:35605)
0810 14:35:46.479073 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.479078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.480521 (+ 1443us) server_negotiation.cc:407] Connection header received
0810 14:35:46.482141 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.482144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.482179 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.482256 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.483343 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.483346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.483348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.483359 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.483383 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.483391 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.483473 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.483599 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.483637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:35:46.484635 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.479494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35605 (local address 127.21.43.67:37843)
0810 14:35:46.480340 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.480395 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.481935 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.482417 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.482423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.482508 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.483040 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.483042 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.483082 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.483186 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.483474 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.483480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.483485 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.483613 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.483805 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":150,"threads_started":1}
I20260810 14:35:46.485824 27258 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35605
I20260810 14:35:46.486193 27258 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:46.486725 27258 heartbeater.cc:511] Master 127.21.43.126:35605 requested a full tablet report, sending...
I20260810 14:35:46.488050 26761 ts_manager.cc:194] Registered new tserver with Master: 4c83cfdc2df7448994d6f440a2330308 (127.21.43.67:44403)
I20260810 14:35:46.493764 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:46.497786 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.495892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.21.43.126:35605)
0810 14:35:46.496067 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.496071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.496341 (+ 270us) server_negotiation.cc:407] Connection header received
0810 14:35:46.496495 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.496498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.496536 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.496628 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.497362 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.497365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.497367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.497379 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.497399 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.497408 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.497507 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.497585 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.497631 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:46.500900 26761 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:46.504516 26761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:46.516891 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.513593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:33331 (local address 127.0.0.1:35280)
0810 14:35:46.514363 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.514379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.514451 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.515223 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.515229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.515316 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.516043 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.516045 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.516064 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.516154 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.516502 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.516512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.516518 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.516630 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.516703 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":133,"threads_started":1}
I20260810 14:35:46.517187 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.513477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:44403 (local address 127.0.0.1:38350)
0810 14:35:46.513976 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.514043 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.514156 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.515223 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.515229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.515309 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.516126 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.516128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.516136 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.516153 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.516680 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.516684 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.516687 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.516745 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.516782 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":84,"threads_started":1}
I20260810 14:35:46.517519 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.514286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.21.43.65:33331)
0810 14:35:46.514724 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.514738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.514762 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:35:46.514820 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.514828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.515029 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.515205 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.516316 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.516330 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.516340 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.516351 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.516388 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.516410 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.516533 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.517277 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.517365 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":136,"threads_started":1}
I20260810 14:35:46.517663 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.513618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.21.43.67:44403)
0810 14:35:46.514724 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.514739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.514762 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:35:46.514858 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.514863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.515028 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.515205 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.516458 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.516464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.516474 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.516485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.516524 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.516569 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.517385 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.517437 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.517507 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":169,"threads_started":1}
I20260810 14:35:46.518890 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.514237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:43231 (local address 127.0.0.1:57492)
0810 14:35:46.516006 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.516022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.516120 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.516619 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.516622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.516638 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.517106 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.517108 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.517120 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.518059 (+ 939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.518645 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.518649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.518652 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.518742 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.518778 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1692,"thread_start_us":102,"threads_started":1}
I20260810 14:35:46.519011 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.514467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.21.43.66:43231)
0810 14:35:46.515087 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.515096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.516243 (+ 1147us) server_negotiation.cc:407] Connection header received
0810 14:35:46.516304 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.516309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.516499 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.516646 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.518469 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.518474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.518489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.518498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.518532 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.518549 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.518635 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.518786 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.518854 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":153,"threads_started":1}
I20260810 14:35:46.520674 27185 tablet_service.cc:1467] Processing CreateTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 (DEFAULT_TABLE table=test-table [id=8b40f7175bd04e2bbef05a713692d49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:46.520634 26892 tablet_service.cc:1467] Processing CreateTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 (DEFAULT_TABLE table=test-table [id=8b40f7175bd04e2bbef05a713692d49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:46.521742 26892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e702b292d3f432c94f7c88f43a53ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:46.521778 27185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e702b292d3f432c94f7c88f43a53ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:46.522125 27037 tablet_service.cc:1467] Processing CreateTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 (DEFAULT_TABLE table=test-table [id=8b40f7175bd04e2bbef05a713692d49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:46.523180 27037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e702b292d3f432c94f7c88f43a53ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:46.528294 27276 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Bootstrap starting.
I20260810 14:35:46.529597 27277 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Bootstrap starting.
I20260810 14:35:46.530170 27278 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Bootstrap starting.
I20260810 14:35:46.530814 27276 tablet_bootstrap.cc:654] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:46.531814 27276 log.cc:824] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:46.531975 27277 tablet_bootstrap.cc:654] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:46.532518 27278 tablet_bootstrap.cc:654] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:46.532850 27277 log.cc:824] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:46.533427 27278 log.cc:824] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:46.534245 27276 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: No bootstrap required, opened a new log
I20260810 14:35:46.534472 27276 ts_tablet_manager.cc:1397] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 14:35:46.534811 27277 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: No bootstrap required, opened a new log
I20260810 14:35:46.535020 27277 ts_tablet_manager.cc:1397] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:46.535490 27278 tablet_bootstrap.cc:492] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: No bootstrap required, opened a new log
I20260810 14:35:46.535693 27278 ts_tablet_manager.cc:1397] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:35:46.539047 27277 raft_consensus.cc:348] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.539400 27277 raft_consensus.cc:374] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:46.539460 27277 raft_consensus.cc:729] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c83cfdc2df7448994d6f440a2330308, State: Initialized, Role: FOLLOWER
I20260810 14:35:46.539512 27276 raft_consensus.cc:348] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.539772 27276 raft_consensus.cc:374] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:46.539844 27276 raft_consensus.cc:729] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f609a4e51a4f4cb4b037aeae2f07d3, State: Initialized, Role: FOLLOWER
I20260810 14:35:46.540014 27277 consensus_queue.cc:260] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.540280 27276 consensus_queue.cc:260] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.540335 27278 raft_consensus.cc:348] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.540584 27278 raft_consensus.cc:374] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:46.540659 27278 raft_consensus.cc:729] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4b617d154b40f59092662529d50e96, State: Initialized, Role: FOLLOWER
I20260810 14:35:46.540915 27258 heartbeater.cc:503] Master 127.21.43.126:35605 was elected leader, sending a full tablet report...
I20260810 14:35:46.541057 27278 consensus_queue.cc:260] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.541374 27276 ts_tablet_manager.cc:1428] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 14:35:46.541704 26967 heartbeater.cc:503] Master 127.21.43.126:35605 was elected leader, sending a full tablet report...
I20260810 14:35:46.542088 27110 heartbeater.cc:503] Master 127.21.43.126:35605 was elected leader, sending a full tablet report...
I20260810 14:35:46.542750 27278 ts_tablet_manager.cc:1428] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:35:46.543458 27277 ts_tablet_manager.cc:1428] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260810 14:35:46.601953 26969 tablet.cc:2369] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:46.631647 27283 raft_consensus.cc:482] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:46.631968 27283 raft_consensus.cc:504] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.633216 27283 leader_election.cc:288] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6f609a4e51a4f4cb4b037aeae2f07d3 (127.21.43.65:33331), 9d4b617d154b40f59092662529d50e96 (127.21.43.66:43231)
I20260810 14:35:46.635285 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.633652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:52661 (local address 127.21.43.65:33331)
0810 14:35:46.633785 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.633789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.633805 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:46.633944 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.633947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.633982 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.634082 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.634871 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.634874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.634876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.634886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.634902 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.634910 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.635007 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.635064 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.635104 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:46.635285 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.633574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:33331 (local address 127.21.43.67:52661)
0810 14:35:46.633712 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.633728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.633823 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.634106 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.634109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.634125 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.634702 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.634703 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.634713 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.634736 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.634990 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.634993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.634997 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.635068 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.635104 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:35:46.635530 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.633630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:38403 (local address 127.21.43.66:43231)
0810 14:35:46.633773 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.633778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.634217 (+ 439us) server_negotiation.cc:407] Connection header received
0810 14:35:46.634300 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.634306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.634351 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.634421 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.635057 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.635061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.635064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.635076 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.635099 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.635113 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.635267 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.635333 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.635380 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:35:46.635726 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.633574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:43231 (local address 127.21.43.67:38403)
0810 14:35:46.634128 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.634140 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:46.634228 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:46.634464 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:46.634468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:46.634484 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:46.634889 (+ 405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:35:46.634890 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:46.634899 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:35:46.634923 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:46.635250 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:46.635253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:46.635256 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:35:46.635458 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 14:35:46.635492 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":147,"thread_start_us":180,"threads_started":1}
I20260810 14:35:46.635988 26914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e702b292d3f432c94f7c88f43a53ad6" candidate_uuid: "4c83cfdc2df7448994d6f440a2330308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" is_pre_election: true
I20260810 14:35:46.636353 26914 raft_consensus.cc:2454] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c83cfdc2df7448994d6f440a2330308 in term 0.
I20260810 14:35:46.636343 27061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e702b292d3f432c94f7c88f43a53ad6" candidate_uuid: "4c83cfdc2df7448994d6f440a2330308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d4b617d154b40f59092662529d50e96" is_pre_election: true
I20260810 14:35:46.636606 27061 raft_consensus.cc:2454] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c83cfdc2df7448994d6f440a2330308 in term 0.
I20260810 14:35:46.636857 27131 leader_election.cc:302] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c83cfdc2df7448994d6f440a2330308, b6f609a4e51a4f4cb4b037aeae2f07d3; no voters:
I20260810 14:35:46.637151 27283 raft_consensus.cc:2781] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:46.637312 27283 raft_consensus.cc:482] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:46.637387 27283 raft_consensus.cc:3037] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:46.638711 27283 raft_consensus.cc:504] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.639127 27283 leader_election.cc:288] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [CANDIDATE]: Term 1 election: Requested vote from peers b6f609a4e51a4f4cb4b037aeae2f07d3 (127.21.43.65:33331), 9d4b617d154b40f59092662529d50e96 (127.21.43.66:43231)
I20260810 14:35:46.639614 27061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e702b292d3f432c94f7c88f43a53ad6" candidate_uuid: "4c83cfdc2df7448994d6f440a2330308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d4b617d154b40f59092662529d50e96"
I20260810 14:35:46.639765 27061 raft_consensus.cc:3037] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:46.640874 27061 raft_consensus.cc:2454] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c83cfdc2df7448994d6f440a2330308 in term 1.
I20260810 14:35:46.641219 26914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e702b292d3f432c94f7c88f43a53ad6" candidate_uuid: "4c83cfdc2df7448994d6f440a2330308" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3"
I20260810 14:35:46.641319 27134 leader_election.cc:302] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c83cfdc2df7448994d6f440a2330308, 9d4b617d154b40f59092662529d50e96; no voters:
I20260810 14:35:46.641376 26914 raft_consensus.cc:3037] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:46.641512 27283 raft_consensus.cc:2781] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:46.641760 27283 raft_consensus.cc:686] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 1 LEADER]: Becoming Leader. State: Replica: 4c83cfdc2df7448994d6f440a2330308, State: Running, Role: LEADER
I20260810 14:35:46.642179 27283 consensus_queue.cc:237] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } }
I20260810 14:35:46.642822 26914 raft_consensus.cc:2454] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c83cfdc2df7448994d6f440a2330308 in term 1.
I20260810 14:35:46.645982 26761 catalog_manager.cc:5495] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c83cfdc2df7448994d6f440a2330308 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "4c83cfdc2df7448994d6f440a2330308" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c83cfdc2df7448994d6f440a2330308" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:46.681833 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.679511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.21.43.67:44403)
0810 14:35:46.679653 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.679656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.679670 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:35:46.679753 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.679755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.679826 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.679919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.681406 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.681409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.681411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.681424 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.681455 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.681472 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.681578 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.681668 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.681709 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:35:46.688611 26914 raft_consensus.cc:1261] T 4e702b292d3f432c94f7c88f43a53ad6 P b6f609a4e51a4f4cb4b037aeae2f07d3 [term 1 FOLLOWER]: Refusing update from remote peer 4c83cfdc2df7448994d6f440a2330308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:46.688611 27061 raft_consensus.cc:1261] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 1 FOLLOWER]: Refusing update from remote peer 4c83cfdc2df7448994d6f440a2330308: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:46.689393 27283 consensus_queue.cc:1035] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d4b617d154b40f59092662529d50e96" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:46.689657 27288 consensus_queue.cc:1035] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6f609a4e51a4f4cb4b037aeae2f07d3" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:46.697578 27291 mvcc.cc:205] Tried to move back new op lower bound from 7316981951228264448 to 7316981951050153984. Current Snapshot: MvccSnapshot[applied={T|T < 7316981951228264448}]
I20260810 14:35:46.700558 27292 mvcc.cc:205] Tried to move back new op lower bound from 7316981951228264448 to 7316981951050153984. Current Snapshot: MvccSnapshot[applied={T|T < 7316981951228264448}]
I20260810 14:35:46.700789 27293 mvcc.cc:205] Tried to move back new op lower bound from 7316981951228264448 to 7316981951050153984. Current Snapshot: MvccSnapshot[applied={T|T < 7316981951228264448 or (T in {7316981951228264448})}]
I20260810 14:35:46.707819 26759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:46.708030 26759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:46.710835 26759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e83788a4f9bc4a7897ea1501ddd4cf40: Sending DeleteTablet for 3 replicas of tablet 4e702b292d3f432c94f7c88f43a53ad6
I20260810 14:35:46.711797 27037 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:46 UTC) from {username='slave'} at 127.0.0.1:57492
I20260810 14:35:46.711779 26892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:46 UTC) from {username='slave'} at 127.0.0.1:35280
I20260810 14:35:46.711862 27185 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e702b292d3f432c94f7c88f43a53ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:46 UTC) from {username='slave'} at 127.0.0.1:38350
I20260810 14:35:46.712383 27301 tablet_replica.cc:330] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: stopping tablet replica
I20260810 14:35:46.712405 27303 tablet_replica.cc:330] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: stopping tablet replica
I20260810 14:35:46.712764 27303 raft_consensus.cc:2229] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:46.712787 27301 raft_consensus.cc:2229] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:46.712914 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26827
I20260810 14:35:46.713240 27303 raft_consensus.cc:2258] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:46.713265 27301 raft_consensus.cc:2258] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:46.716250 27301 ts_tablet_manager.cc:1905] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:46.719348 27301 ts_tablet_manager.cc:1918] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:46.719544 27301 log.cc:1196] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/4e702b292d3f432c94f7c88f43a53ad6
I20260810 14:35:46.719985 27301 ts_tablet_manager.cc:1939] T 4e702b292d3f432c94f7c88f43a53ad6 P 9d4b617d154b40f59092662529d50e96: Deleting consensus metadata
I20260810 14:35:46.721542 26747 catalog_manager.cc:4841] TS 9d4b617d154b40f59092662529d50e96 (127.21.43.66:43231): tablet 4e702b292d3f432c94f7c88f43a53ad6 (table test-table [id=8b40f7175bd04e2bbef05a713692d49a]) successfully deleted
I20260810 14:35:46.724397 27303 ts_tablet_manager.cc:1905] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:46.726931 27303 ts_tablet_manager.cc:1918] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:46.727077 27303 log.cc:1196] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/4e702b292d3f432c94f7c88f43a53ad6
I20260810 14:35:46.727499 27303 ts_tablet_manager.cc:1939] T 4e702b292d3f432c94f7c88f43a53ad6 P 4c83cfdc2df7448994d6f440a2330308: Deleting consensus metadata
I20260810 14:35:46.728989 26747 catalog_manager.cc:4841] TS 4c83cfdc2df7448994d6f440a2330308 (127.21.43.67:44403): tablet 4e702b292d3f432c94f7c88f43a53ad6 (table test-table [id=8b40f7175bd04e2bbef05a713692d49a]) successfully deleted
W20260810 14:35:46.731976 26744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:33331 (error 108)
I20260810 14:35:46.732074 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26971
I20260810 14:35:46.735350 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.734852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:33331 (local address 127.0.0.1:35290)
0810 14:35:46.735127 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:46.735237 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260810 14:35:46.735544 26744 catalog_manager.cc:4584] TS b6f609a4e51a4f4cb4b037aeae2f07d3 (127.21.43.65:33331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e702b292d3f432c94f7c88f43a53ad6: Network error: Client connection negotiation failed: client connection to 127.21.43.65:33331: connect: Connection refused (error 111)
I20260810 14:35:46.743424 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27113
I20260810 14:35:46.752233 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 26727
2026-08-10T14:35:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1161 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 14:35:46.778188 21677 mini_oidc.cc:152] Starting JWKS server
I20260810 14:35:46.780093 21677 webserver.cc:460] Webserver started at https://127.0.0.1:38035/ using document root <none> and password file <none>
I20260810 14:35:46.780303 21677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:35:46.780588 21677 webserver.cc:460] Webserver started at http://127.0.0.1:38597/ using document root <none> and password file <none>
2026-08-10T14:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:46Z Disabled control of system clock
I20260810 14:35:46.791690 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:36233
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:36233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/testchainca.cert with env {}
W20260810 14:35:46.907795 27310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:46.908135 27310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:46.908179 27310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:46.912389 27310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:46.912470 27310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:46.912487 27310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:46.912503 27310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:46.912519 27310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:35:46.912535 27310 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:35:46.912549 27310 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38035/jwks.json
I20260810 14:35:46.918380 27310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:36233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:46.919531 27310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:46.920565 27310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:46.929791 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.929811 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:46.929740 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:46.930090 27310 server_base.cc:1034] running on GCE node
I20260810 14:35:46.930702 27310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:46.931960 27310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:46.933197 27310 hybrid_clock.cc:648] HybridClock initialized: now 1786372546933170 us; error 48 us; skew 500 ppm
I20260810 14:35:46.935460 27310 webserver.cc:460] Webserver started at http://127.21.43.126:35513/ using document root <none> and password file <none>
I20260810 14:35:46.936100 27310 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:46.936201 27310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:46.936448 27310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:46.938673 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "d4fcba53a8904a13824f60235c03d457"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "238c5aa885c6535512b41f8b665908a8"
server_key_iv: "811d602052abd1b8d671613a1ee645f2"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.939343 27310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "d4fcba53a8904a13824f60235c03d457"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "238c5aa885c6535512b41f8b665908a8"
server_key_iv: "811d602052abd1b8d671613a1ee645f2"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.943405 27310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:46.946012 27327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:46.947434 27310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:46.947599 27310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "d4fcba53a8904a13824f60235c03d457"
format_stamp: "Formatted at 2026-08-10 14:35:46 on dist-test-slave-chsk"
server_key: "238c5aa885c6535512b41f8b665908a8"
server_key_iv: "811d602052abd1b8d671613a1ee645f2"
server_key_version: "encryptionkey@0"
I20260810 14:35:46.947728 27310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:46.969324 27310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:46.970129 27310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:46.970352 27310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:46.977581 27310 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:36233
I20260810 14:35:46.977596 27384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:36233 every 8 connection(s)
I20260810 14:35:46.978591 27310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:46.979365 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27310
I20260810 14:35:46.979542 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:46.979879 21677 external_mini_cluster.cc:1428] Setting key 09a67082afec797f389e35a14c732282
I20260810 14:35:46.982600 27386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:46.988626 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457: Bootstrap starting.
I20260810 14:35:46.989674 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:46.981371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.21.43.126:36233)
0810 14:35:46.982124 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:46.982142 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:46.982180 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:35:46.982895 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:46.982913 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:46.983404 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:46.983745 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:46.984183 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:46.985169 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:46.986121 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:46.986605 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:46.986782 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:46.987789 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:46.987793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:46.987807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:46.988235 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:46.988295 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:46.988316 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:46.988430 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:46.988596 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 14:35:46.988898 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":134,"threads_started":1}
I20260810 14:35:46.991266 27386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:46.992187 27386 log.cc:824] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:46.992930 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:35:46.994536 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457: No bootstrap required, opened a new log
I20260810 14:35:46.998520 27386 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } }
I20260810 14:35:46.998783 27386 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:46.998862 27386 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fcba53a8904a13824f60235c03d457, State: Initialized, Role: FOLLOWER
I20260810 14:35:46.999390 27386 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } }
I20260810 14:35:46.999523 27386 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:46.999611 27386 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:46.999742 27386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:47.000756 27386 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } }
I20260810 14:35:47.001147 27386 leader_election.cc:302] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4fcba53a8904a13824f60235c03d457; no voters:
I20260810 14:35:47.001683 27386 leader_election.cc:288] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:47.001806 27391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:47.002099 27391 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [term 1 LEADER]: Becoming Leader. State: Replica: d4fcba53a8904a13824f60235c03d457, State: Running, Role: LEADER
I20260810 14:35:47.002449 27391 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } }
I20260810 14:35:47.003087 27386 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:47.004307 27393 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4fcba53a8904a13824f60235c03d457. Latest consensus state: current_term: 1 leader_uuid: "d4fcba53a8904a13824f60235c03d457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } } }
I20260810 14:35:47.004446 27393 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.004721 27392 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4fcba53a8904a13824f60235c03d457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4fcba53a8904a13824f60235c03d457" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36233 } } }
I20260810 14:35:47.004815 27392 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.004942 27401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:47.008759 27401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:47.015277 27401 catalog_manager.cc:1347] Generated new cluster ID: 8ca8155fb3944fa685aac2d11e2627c2
I20260810 14:35:47.015373 27401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:47.036602 27401 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:47.037938 27401 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:47.057826 27401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4fcba53a8904a13824f60235c03d457: Generated new TSK 0
I20260810 14:35:47.058836 27401 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:35:47.107050 27418 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 14:35:47.107340 27418 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:35:47.107834 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:47.080263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.21.43.126:36233)
0810 14:35:47.080401 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:47.080405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:47.080421 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:47.080646 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:47.080649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:47.080704 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:47.080802 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:35:47.081097 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.081618 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.082913 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.083210 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.083316 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.083783 (+ 467us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:35:47.107307 (+ 23524us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:35:47.107711 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:35:47.107921 27389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 14:35:47.109414 27417 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:47.079867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:36233 (local address 127.0.0.1:47846)
0810 14:35:47.080309 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:47.080326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:47.080458 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:47.080835 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:47.080838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:47.080846 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:35:47.080987 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.080994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.081733 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.081737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.082760 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.082771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.083343 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.083347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.083451 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.083528 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:35:47.107669 (+ 24141us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:35:47.107933 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.43.126:36233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.43.126:36233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 14:35:47.110278 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27310
2026-08-10T14:35:47Z chronyd exiting
I20260810 14:35:47.151757 21677 test_util.cc:175] -----------------------------------------------
I20260810 14:35:47.151872 21677 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372517778458-21677-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (375 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 14:35:47.152971 21677 mini_oidc.cc:152] Starting JWKS server
I20260810 14:35:47.153869 21677 webserver.cc:460] Webserver started at https://127.0.0.1:34723/ using document root <none> and password file <none>
I20260810 14:35:47.154002 21677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:35:47.154304 21677 webserver.cc:460] Webserver started at http://127.0.0.1:45495/ using document root <none> and password file <none>
2026-08-10T14:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:47Z Disabled control of system clock
I20260810 14:35:47.165023 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43729
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43729
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/testchainca.cert with env {}
W20260810 14:35:47.290519 27424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:47.290925 27424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:47.291030 27424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:47.296383 27424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:47.296533 27424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:47.296566 27424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:47.296607 27424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:47.296644 27424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:35:47.296681 27424 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:35:47.296715 27424 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34723/jwks.json
I20260810 14:35:47.302992 27424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:47.304374 27424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:47.305531 27424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:47.315104 27432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.315449 27434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.315104 27430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:47.316125 27424 server_base.cc:1034] running on GCE node
I20260810 14:35:47.316859 27424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:47.318266 27424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:47.319483 27424 hybrid_clock.cc:648] HybridClock initialized: now 1786372547319449 us; error 55 us; skew 500 ppm
I20260810 14:35:47.321867 27424 webserver.cc:460] Webserver started at http://127.21.43.126:42145/ using document root <none> and password file <none>
I20260810 14:35:47.322566 27424 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:47.322680 27424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:47.322937 27424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:47.325316 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "39c2388b4e984dd0a670230dbf114e88"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "5e7b89bb0c262d42da51933b93957b46"
server_key_iv: "b3c0ebf3faf72fa197cfe46cc5dd3009"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.325984 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "39c2388b4e984dd0a670230dbf114e88"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "5e7b89bb0c262d42da51933b93957b46"
server_key_iv: "b3c0ebf3faf72fa197cfe46cc5dd3009"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.330919 27424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:35:47.334028 27441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:47.335624 27424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:47.335804 27424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "39c2388b4e984dd0a670230dbf114e88"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "5e7b89bb0c262d42da51933b93957b46"
server_key_iv: "b3c0ebf3faf72fa197cfe46cc5dd3009"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.335958 27424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:47.346443 27424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:47.348194 27424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:47.348529 27424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:47.357968 27424 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43729
I20260810 14:35:47.357996 27500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43729 every 8 connection(s)
I20260810 14:35:47.359097 27424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:47.362349 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27424
I20260810 14:35:47.362506 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:47.362826 21677 external_mini_cluster.cc:1428] Setting key 7451a391260c0768f07bb911b9bf516c
I20260810 14:35:47.363253 27501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:47.371456 27501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88: Bootstrap starting.
I20260810 14:35:47.373638 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:47.364421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.21.43.126:43729)
0810 14:35:47.365123 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:47.365140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:47.365179 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:35:47.366212 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:47.366236 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:47.366658 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:47.366966 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:47.367436 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.368406 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.369831 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.370344 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.370472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.371607 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:47.371615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:47.371631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:47.372068 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:47.372148 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:47.372171 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:47.372251 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:47.372481 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:35:47.372815 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":170,"threads_started":1}
I20260810 14:35:47.374261 27501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:47.375299 27501 log.cc:824] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:47.377363 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:35:47.377960 27501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88: No bootstrap required, opened a new log
I20260810 14:35:47.382388 27501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } }
I20260810 14:35:47.382678 27501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:47.382720 27501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c2388b4e984dd0a670230dbf114e88, State: Initialized, Role: FOLLOWER
I20260810 14:35:47.383227 27501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } }
I20260810 14:35:47.383373 27501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:47.383431 27501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:47.383524 27501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:47.384603 27501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } }
I20260810 14:35:47.384963 27501 leader_election.cc:302] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c2388b4e984dd0a670230dbf114e88; no voters:
I20260810 14:35:47.385286 27501 leader_election.cc:288] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:47.385444 27507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:47.385753 27507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [term 1 LEADER]: Becoming Leader. State: Replica: 39c2388b4e984dd0a670230dbf114e88, State: Running, Role: LEADER
I20260810 14:35:47.386519 27507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } }
I20260810 14:35:47.386912 27501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:47.388625 27508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c2388b4e984dd0a670230dbf114e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } } }
I20260810 14:35:47.388797 27508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.389180 27510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c2388b4e984dd0a670230dbf114e88. Latest consensus state: current_term: 1 leader_uuid: "39c2388b4e984dd0a670230dbf114e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c2388b4e984dd0a670230dbf114e88" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43729 } } }
I20260810 14:35:47.389333 27510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.389256 27517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:47.392714 27517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:47.399858 27517 catalog_manager.cc:1347] Generated new cluster ID: 08210cc4a44845eabb961758be7920bd
I20260810 14:35:47.399961 27517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:47.407632 27517 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:47.408969 27517 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:47.415027 27517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c2388b4e984dd0a670230dbf114e88: Generated new TSK 0
I20260810 14:35:47.415980 27517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:35:47.451821 27535 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 14:35:47.451936 27535 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:35:47.452610 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:47.428380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.21.43.126:43729)
0810 14:35:47.428535 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:47.428538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:47.428677 (+ 139us) server_negotiation.cc:407] Connection header received
0810 14:35:47.428861 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:47.428863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:47.428921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:47.429033 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:35:47.429446 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.430126 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.431696 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.431945 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.432059 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.432477 (+ 418us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:35:47.452210 (+ 19733us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:35:47.452488 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 14:35:47.452702 27505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 14:35:47.452876 27534 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:47.428186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43729 (local address 127.0.0.1:52506)
0810 14:35:47.428567 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:47.428583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:47.428716 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:47.429055 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:47.429059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:47.429067 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:35:47.429206 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.429214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.430348 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.430353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.431470 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.431483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.432108 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.432113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.432212 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.432250 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:35:47.452460 (+ 20210us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:35:47.452691 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.43.126:43729: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34723/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260810 14:35:47.453605 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27424
2026-08-10T14:35:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 14:35:47.497467 21677 mini_oidc.cc:152] Starting JWKS server
I20260810 14:35:47.498374 21677 webserver.cc:460] Webserver started at https://127.0.0.1:40089/ using document root <none> and password file <none>
I20260810 14:35:47.498512 21677 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:35:47.498816 21677 webserver.cc:460] Webserver started at http://127.0.0.1:35059/ using document root <none> and password file <none>
2026-08-10T14:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:47Z Disabled control of system clock
I20260810 14:35:47.509929 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43641
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43641
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40089/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:35:47.630947 27542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:47.631338 27542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:47.631385 27542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:47.635458 27542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:47.635535 27542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:47.635550 27542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:47.635567 27542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:47.635582 27542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:35:47.635599 27542 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:35:47.635612 27542 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40089/jwks.json
I20260810 14:35:47.640799 27542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40089/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:47.642007 27542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:47.643517 27542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:47.652773 27548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.652813 27547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.652813 27550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:47.653035 27542 server_base.cc:1034] running on GCE node
I20260810 14:35:47.653748 27542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:47.654927 27542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:47.656157 27542 hybrid_clock.cc:648] HybridClock initialized: now 1786372547656156 us; error 56 us; skew 500 ppm
I20260810 14:35:47.658133 27542 webserver.cc:460] Webserver started at http://127.21.43.126:39547/ using document root <none> and password file <none>
I20260810 14:35:47.658735 27542 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:47.658831 27542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:47.659077 27542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:47.661342 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "ae96d9430a6a50afb7fa7d09b264e3a8"
server_key_iv: "567234f4f19c28fcd8fad0aa40f1bf82"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.661947 27542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "ae96d9430a6a50afb7fa7d09b264e3a8"
server_key_iv: "567234f4f19c28fcd8fad0aa40f1bf82"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.665776 27542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 14:35:47.668264 27556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:47.669585 27542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:47.669744 27542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5"
format_stamp: "Formatted at 2026-08-10 14:35:47 on dist-test-slave-chsk"
server_key: "ae96d9430a6a50afb7fa7d09b264e3a8"
server_key_iv: "567234f4f19c28fcd8fad0aa40f1bf82"
server_key_version: "encryptionkey@0"
I20260810 14:35:47.669872 27542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:47.688050 27542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:47.688838 27542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:47.689065 27542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:47.696508 27542 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43641
I20260810 14:35:47.696520 27612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43641 every 8 connection(s)
I20260810 14:35:47.697564 27542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:47.700809 27613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:47.707412 27613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5: Bootstrap starting.
I20260810 14:35:47.707762 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27542
I20260810 14:35:47.707895 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:47.708187 21677 external_mini_cluster.cc:1428] Setting key 84bcf36920407a859dd05723984ec982
I20260810 14:35:47.710448 27613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:47.711345 27613 log.cc:824] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:47.713502 27613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5: No bootstrap required, opened a new log
I20260810 14:35:47.717270 27613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } }
I20260810 14:35:47.717571 27613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:47.717653 27613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4bcca0ecc046fc8b61ebab1c8ad8a5, State: Initialized, Role: FOLLOWER
I20260810 14:35:47.718148 27613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } }
I20260810 14:35:47.718284 27613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:47.718405 27613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:47.718477 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:47.709665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.21.43.126:43641)
0810 14:35:47.710409 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:47.710424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:47.710462 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:35:47.711302 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:47.711323 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:47.711822 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:47.712202 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:47.712677 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.713732 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.714673 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.715234 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.715408 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.716332 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:47.716338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:47.716351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:47.716765 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:47.716822 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:47.716843 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:47.716955 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:47.717131 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 14:35:47.717409 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":159,"threads_started":1}
I20260810 14:35:47.718530 27613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:47.719694 27613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } }
I20260810 14:35:47.720089 27613 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a4bcca0ecc046fc8b61ebab1c8ad8a5; no voters:
I20260810 14:35:47.720489 27613 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:47.720628 27619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:47.720865 27619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [term 1 LEADER]: Becoming Leader. State: Replica: 7a4bcca0ecc046fc8b61ebab1c8ad8a5, State: Running, Role: LEADER
I20260810 14:35:47.721211 27619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } }
I20260810 14:35:47.721678 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:35:47.721917 27613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:47.722530 27619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4bcca0ecc046fc8b61ebab1c8ad8a5. Latest consensus state: current_term: 1 leader_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } } }
I20260810 14:35:47.722648 27619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.723887 27624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:47.724097 27620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bcca0ecc046fc8b61ebab1c8ad8a5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43641 } } }
I20260810 14:35:47.724278 27620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:47.726801 27624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:47.735030 27624 catalog_manager.cc:1347] Generated new cluster ID: 8b58c1259e5347bc87f7f67655688356
I20260810 14:35:47.735160 27624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:47.764958 27624 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:47.766305 27624 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:47.777238 27624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a4bcca0ecc046fc8b61ebab1c8ad8a5: Generated new TSK 0
I20260810 14:35:47.778120 27624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:35:47.801685 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:47.796266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.21.43.126:43641)
0810 14:35:47.796429 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:47.796434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:47.796634 (+ 200us) server_negotiation.cc:407] Connection header received
0810 14:35:47.796823 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:47.796826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:47.796888 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:47.796979 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:35:47.797249 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.797781 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.799099 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.799402 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.799492 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.799853 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:35:47.801236 (+ 1383us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:35:47.801529 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 14:35:47.801789 27617 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 14:35:47.804571 27645 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:47.796183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43641 (local address 127.0.0.1:59824)
0810 14:35:47.796531 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:47.796547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:47.796674 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:47.797006 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:47.797009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:47.797017 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:35:47.797136 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.797142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.797914 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.797918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.798947 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:47.798958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:47.799538 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:47.799542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:47.799635 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:47.799674 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:35:47.801419 (+ 1745us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:35:47.804414 (+ 2995us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.43.126:43641: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":2759,"thread_start_us":134,"threads_started":1}
I20260810 14:35:47.805188 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27542
E20260810 14:35:47.814772 27646 webserver.cc:533] Webserver: sslize failed:
E20260810 14:35:47.814884 27646 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T14:35:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T14:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:47Z Disabled control of system clock
I20260810 14:35:47.858659 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:45475
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:45475
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/testchainca.cert with env {}
W20260810 14:35:47.976575 27652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:47.976874 27652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:47.976917 27652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:47.981165 27652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:47.981240 27652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:47.981254 27652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:47.981272 27652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:47.981287 27652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:35:47.981331 27652 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:35:47.981361 27652 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 14:35:47.987383 27652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:45475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:45475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:47.988557 27652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:47.989612 27652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:47.998778 27660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.998780 27662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:47.998778 27658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:47.999405 27652 server_base.cc:1034] running on GCE node
I20260810 14:35:48.000037 27652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:48.001284 27652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:48.002511 27652 hybrid_clock.cc:648] HybridClock initialized: now 1786372548002484 us; error 47 us; skew 500 ppm
I20260810 14:35:48.004840 27652 webserver.cc:460] Webserver started at http://127.21.43.126:33129/ using document root <none> and password file <none>
I20260810 14:35:48.005544 27652 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:48.005642 27652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:48.005878 27652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:48.008153 27652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "2f51cd4825604b9fa33734316e9dcdfd"
format_stamp: "Formatted at 2026-08-10 14:35:48 on dist-test-slave-chsk"
server_key: "7aebc9b0bb9e324654367deec37a1221"
server_key_iv: "925c4eab69a0b39e1899cc355a4eaa2f"
server_key_version: "encryptionkey@0"
I20260810 14:35:48.008798 27652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "2f51cd4825604b9fa33734316e9dcdfd"
format_stamp: "Formatted at 2026-08-10 14:35:48 on dist-test-slave-chsk"
server_key: "7aebc9b0bb9e324654367deec37a1221"
server_key_iv: "925c4eab69a0b39e1899cc355a4eaa2f"
server_key_version: "encryptionkey@0"
I20260810 14:35:48.012991 27652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 14:35:48.015870 27668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:48.017294 27652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:48.017462 27652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "2f51cd4825604b9fa33734316e9dcdfd"
format_stamp: "Formatted at 2026-08-10 14:35:48 on dist-test-slave-chsk"
server_key: "7aebc9b0bb9e324654367deec37a1221"
server_key_iv: "925c4eab69a0b39e1899cc355a4eaa2f"
server_key_version: "encryptionkey@0"
I20260810 14:35:48.017596 27652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:48.036278 27652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:48.037246 27652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:48.037467 27652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:48.047364 27652 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:45475
I20260810 14:35:48.047375 27725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:45475 every 8 connection(s)
I20260810 14:35:48.048794 27652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:48.053078 27726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:48.055739 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27652
I20260810 14:35:48.055873 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:48.056216 21677 external_mini_cluster.cc:1428] Setting key 50c1e39a91b4186c7e1c57c4e950380b
I20260810 14:35:48.059772 27726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd: Bootstrap starting.
I20260810 14:35:48.062605 27726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:48.063627 27726 log.cc:824] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:48.066287 27726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd: No bootstrap required, opened a new log
I20260810 14:35:48.069104 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:48.057681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.21.43.126:45475)
0810 14:35:48.058386 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:48.058411 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:48.058460 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:35:48.059485 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:48.059510 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:48.060068 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:48.060485 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:48.060999 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.062688 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.064171 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.064937 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.065135 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:48.066140 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:48.066171 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:48.066192 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:35:48.066780 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:48.066869 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:35:48.066898 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:48.067010 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:48.067207 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 14:35:48.067690 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260810 14:35:48.069715 27726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } }
I20260810 14:35:48.069980 27726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:48.070043 27726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f51cd4825604b9fa33734316e9dcdfd, State: Initialized, Role: FOLLOWER
I20260810 14:35:48.070585 27726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } }
I20260810 14:35:48.070749 27726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:48.070812 27726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:48.070914 27726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:48.072065 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:35:48.072330 27726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } }
I20260810 14:35:48.072742 27726 leader_election.cc:302] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f51cd4825604b9fa33734316e9dcdfd; no voters:
I20260810 14:35:48.073134 27726 leader_election.cc:288] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:48.073252 27732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:48.073482 27732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [term 1 LEADER]: Becoming Leader. State: Replica: 2f51cd4825604b9fa33734316e9dcdfd, State: Running, Role: LEADER
I20260810 14:35:48.073796 27732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } }
I20260810 14:35:48.074611 27726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:48.075028 27732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f51cd4825604b9fa33734316e9dcdfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } } }
I20260810 14:35:48.075198 27732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:48.075420 27732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f51cd4825604b9fa33734316e9dcdfd. Latest consensus state: current_term: 1 leader_uuid: "2f51cd4825604b9fa33734316e9dcdfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51cd4825604b9fa33734316e9dcdfd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 45475 } } }
I20260810 14:35:48.075513 27732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:48.075837 27736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:48.079161 27736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:48.086203 27736 catalog_manager.cc:1347] Generated new cluster ID: 996341f7dd0142d99503c0c7ef5c4db6
I20260810 14:35:48.086294 27736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:48.121819 27736 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:48.123216 27736 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:48.130786 27736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f51cd4825604b9fa33734316e9dcdfd: Generated new TSK 0
I20260810 14:35:48.131573 27736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:35:48.590950 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:48.147111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.21.43.126:45475)
0810 14:35:48.147278 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:48.147283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:48.147423 (+ 140us) server_negotiation.cc:407] Connection header received
0810 14:35:48.147579 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:48.147583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:48.147645 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:48.147770 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:35:48.147978 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.148807 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.149666 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.150022 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.150134 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:48.150426 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:35:48.590417 (+439991us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:35:48.590784 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36116: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 14:35:48.591074 27729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36116: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 14:35:48.591110 27756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:48.147011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:45475 (local address 127.0.0.1:36116)
0810 14:35:48.147334 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:48.147347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:48.147443 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:48.147769 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:48.147771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:48.147776 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:35:48.147883 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:48.147889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.148929 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.148931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:48.149554 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:48.149561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:48.150149 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:48.150152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:48.150222 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:48.150245 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:35:48.590682 (+440437us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:35:48.590887 (+ 205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.43.126:45475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20260810 14:35:48.592015 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27652
2026-08-10T14:35:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (131 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:48 dist-test-slave-chsk krb5kdc[27769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:48 dist-test-slave-chsk krb5kdc[27769](info): set up 2 sockets
Aug 10 14:35:48 dist-test-slave-chsk krb5kdc[27769](info): commencing operation
krb5kdc: starting...
W20260810 14:35:49.979662 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:35:50.122841 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41093
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:50.244758 27783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:50.245069 27783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:50.245113 27783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:50.249348 27783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:50.249431 27783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:50.249452 27783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:50.249469 27783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:50.249485 27783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:50.255162 27783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:50.256366 27783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:50.257464 27783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:35:50.266739 27783 server_base.cc:1034] running on GCE node
W20260810 14:35:50.266819 27791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.266945 27789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.266819 27788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:50.267685 27783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:50.268919 27783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:50.270097 27783 hybrid_clock.cc:648] HybridClock initialized: now 1786372550270079 us; error 35 us; skew 500 ppm
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:50.277383 27783 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:35:50.278466 27783 webserver.cc:460] Webserver started at http://127.21.43.126:39435/ using document root <none> and password file <none>
I20260810 14:35:50.279031 27783 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:50.279095 27783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:50.279352 27783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:50.281469 27783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "410dd54c5f114abd8aa66e942716e221"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "f4e7d18aeb5cc110c12fb27daa4a745d"
server_key_iv: "07f5a7c432199197d23a6a473dc4cf9e"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.282028 27783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "410dd54c5f114abd8aa66e942716e221"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "f4e7d18aeb5cc110c12fb27daa4a745d"
server_key_iv: "07f5a7c432199197d23a6a473dc4cf9e"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.286029 27783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:50.288964 27799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.290766 27783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:50.290952 27783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "410dd54c5f114abd8aa66e942716e221"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "f4e7d18aeb5cc110c12fb27daa4a745d"
server_key_iv: "07f5a7c432199197d23a6a473dc4cf9e"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.291090 27783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:50.307742 27783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:50.311571 27783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:50.311826 27783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:50.319408 27783 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:41093
I20260810 14:35:50.319494 27857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:41093 every 8 connection(s)
I20260810 14:35:50.320495 27783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:50.321811 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27783
I20260810 14:35:50.321954 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:50.322288 21677 external_mini_cluster.cc:1428] Setting key decdfba0c176eb3aeb05985780605e77
I20260810 14:35:50.324047 27858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:50.330293 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221: Bootstrap starting.
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:50.333142 27858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:50.334038 27858 log.cc:824] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:50.336689 27858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221: No bootstrap required, opened a new log
I20260810 14:35:50.337711 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:50.323857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.21.43.126:41093)
0810 14:35:50.324395 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:50.324411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:50.324449 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:35:50.325098 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:50.325115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:50.325459 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:50.325842 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:50.326754 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.327859 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.328760 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.329269 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.329448 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:50.332402 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:50.332408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:50.332422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:50.332454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:50.334309 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.334839 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.334844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.334849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.335005 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.335388 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.335391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.335392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.335923 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:50.336149 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:50.336417 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 14:35:50.336737 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":173,"threads_started":1}
I20260810 14:35:50.340960 27858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } }
I20260810 14:35:50.341202 27858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:50.341253 27858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410dd54c5f114abd8aa66e942716e221, State: Initialized, Role: FOLLOWER
I20260810 14:35:50.341733 27858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } }
I20260810 14:35:50.341898 27858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:50.342008 27858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:50.342128 27858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:50.343212 27858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } }
I20260810 14:35:50.343607 27858 leader_election.cc:302] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 410dd54c5f114abd8aa66e942716e221; no voters:
I20260810 14:35:50.343991 27858 leader_election.cc:288] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:50.344173 27863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:50.344488 27863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [term 1 LEADER]: Becoming Leader. State: Replica: 410dd54c5f114abd8aa66e942716e221, State: Running, Role: LEADER
I20260810 14:35:50.344831 27863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } }
I20260810 14:35:50.345332 27858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:50.346606 27864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "410dd54c5f114abd8aa66e942716e221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } } }
I20260810 14:35:50.346748 27864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:50.347162 27865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 410dd54c5f114abd8aa66e942716e221. Latest consensus state: current_term: 1 leader_uuid: "410dd54c5f114abd8aa66e942716e221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "410dd54c5f114abd8aa66e942716e221" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41093 } } }
I20260810 14:35:50.347261 27865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:50.347561 27873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:50.350782 27873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:50.356770 27873 catalog_manager.cc:1347] Generated new cluster ID: 86f347bf171b481b9ddcb1fee51244c5
I20260810 14:35:50.356864 27873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:50.368506 27873 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:50.369654 27873 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:50.379055 27873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 410dd54c5f114abd8aa66e942716e221: Generated new TSK 0
I20260810 14:35:50.380044 27873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:35:50.426935 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:50.552947 27885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:50.553303 27885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:50.553386 27885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:50.557684 27885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:50.557788 27885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:50.557888 27885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:50.563748 27885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:50.564926 27885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:50.566020 27885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:50.576514 27892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.576511 27891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.576529 27894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:50.576920 27885 server_base.cc:1034] running on GCE node
I20260810 14:35:50.577353 27885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:50.578066 27885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:50.579247 27885 hybrid_clock.cc:648] HybridClock initialized: now 1786372550579233 us; error 29 us; skew 500 ppm
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:50.582307 27885 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:35:50.583384 27885 webserver.cc:460] Webserver started at http://127.21.43.65:42709/ using document root <none> and password file <none>
I20260810 14:35:50.583952 27885 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:50.584015 27885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:50.584190 27885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:50.586325 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "64bfcf27bdf845fdb2dc56738974d09a"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "a018cf962b00b091bf1fed155f759fba"
server_key_iv: "dfc74fa51eda1928440410867200e873"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.586959 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "64bfcf27bdf845fdb2dc56738974d09a"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "a018cf962b00b091bf1fed155f759fba"
server_key_iv: "dfc74fa51eda1928440410867200e873"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.590935 27885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:50.593708 27903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.595386 27885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:50.595530 27885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "64bfcf27bdf845fdb2dc56738974d09a"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "a018cf962b00b091bf1fed155f759fba"
server_key_iv: "dfc74fa51eda1928440410867200e873"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.595659 27885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:50.611608 27885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:50.615350 27885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:50.615589 27885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:50.616273 27885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:50.617200 27885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:50.617276 27885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.617355 27885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:50.617400 27885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.627640 27885 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:38477
I20260810 14:35:50.627671 28026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:38477 every 8 connection(s)
I20260810 14:35:50.628664 27885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:50.636252 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 27885
I20260810 14:35:50.636368 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:50.636708 21677 external_mini_cluster.cc:1428] Setting key 8a32e5bc012a9abb9535c73f755fb590
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:50.645042 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:50.630841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:59099 (local address 127.21.43.126:41093)
0810 14:35:50.631011 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:50.631015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:50.631888 (+ 873us) server_negotiation.cc:407] Connection header received
0810 14:35:50.633430 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:50.633434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:50.633509 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:50.633620 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:50.634931 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.635593 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.636937 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.637264 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.637380 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:50.640619 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:50.640622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:50.640624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:50.640657 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:50.642209 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.642960 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.642964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.642966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.643060 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.643549 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.643556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.643560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.643847 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:50.643971 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:50.644579 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 14:35:50.644755 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:35:50.645821 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:50.631117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41093 (local address 127.21.43.65:59099)
0810 14:35:50.631718 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:50.631763 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:50.633167 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:50.633823 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:50.633829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:50.634224 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:50.634751 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:50.634766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.635743 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.635753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:50.636792 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:50.636807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.637475 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.637480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:50.637701 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:50.638588 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:50.638611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:50.640416 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:50.642464 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:50.642471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:50.642490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:50.642837 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:50.643186 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:50.643192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:50.643194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:50.643404 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:50.643974 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:50.643980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:50.644268 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:35:50.644436 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 14:35:50.644738 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":138,"threads_started":1}
I20260810 14:35:50.647243 28027 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41093
I20260810 14:35:50.647562 28027 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:50.648227 28027 heartbeater.cc:511] Master 127.21.43.126:41093 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:50.650048 27817 ts_manager.cc:194] Registered new tserver with Master: 64bfcf27bdf845fdb2dc56738974d09a (127.21.43.65:38477)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:50.651528 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:59099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:35:50.669708 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:50.794432 28032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:50.794770 28032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:50.794849 28032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:50.799283 28032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:50.799355 28032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:50.799436 28032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:50.805115 28032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:50.806347 28032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:50.807492 28032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:50.818512 28040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.818506 28038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:50.818506 28037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:50.818697 28032 server_base.cc:1034] running on GCE node
I20260810 14:35:50.819598 28032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:50.820390 28032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:50.821594 28032 hybrid_clock.cc:648] HybridClock initialized: now 1786372550821558 us; error 49 us; skew 500 ppm
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:50.824761 28032 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:35:50.825721 28032 webserver.cc:460] Webserver started at http://127.21.43.66:41381/ using document root <none> and password file <none>
I20260810 14:35:50.826279 28032 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:50.826342 28032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:50.826527 28032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:50.828863 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "655a8d88443244a698d72dd2e0741960"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "4f77aafa9602cf425a4022413a54fbb6"
server_key_iv: "22e39fd35c34a899f9d6486b4ab46be4"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.829456 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "655a8d88443244a698d72dd2e0741960"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "4f77aafa9602cf425a4022413a54fbb6"
server_key_iv: "22e39fd35c34a899f9d6486b4ab46be4"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.834093 28032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:50.837121 28048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.838899 28032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:50.839084 28032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "655a8d88443244a698d72dd2e0741960"
format_stamp: "Formatted at 2026-08-10 14:35:50 on dist-test-slave-chsk"
server_key: "4f77aafa9602cf425a4022413a54fbb6"
server_key_iv: "22e39fd35c34a899f9d6486b4ab46be4"
server_key_version: "encryptionkey@0"
I20260810 14:35:50.839284 28032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:50.857105 28032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:50.860786 28032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:50.861018 28032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:50.861883 28032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:50.862891 28032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:50.862967 28032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.863044 28032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:50.863095 28032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:50.874424 28032 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:38285
I20260810 14:35:50.874444 28171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:38285 every 8 connection(s)
I20260810 14:35:50.875489 28032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:50.878827 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28032
I20260810 14:35:50.879004 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:50.879508 21677 external_mini_cluster.cc:1428] Setting key 655d80d0bc28e568706a086b107ed19c
Aug 10 14:35:50 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372550, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:35:50.893139 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:50.878210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:52081 (local address 127.21.43.126:41093)
0810 14:35:50.878384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:50.878388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:50.879845 (+ 1457us) server_negotiation.cc:407] Connection header received
0810 14:35:50.881653 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:50.881655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:50.881703 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:50.881783 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:50.883600 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.884448 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.885690 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.885931 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.886050 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:50.888993 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:50.888996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:50.888999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:50.889037 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:50.890631 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.891259 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.891262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.891263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.891347 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:50.891715 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:50.891718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:50.891720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:50.891931 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:50.892093 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:50.892800 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 14:35:50.892968 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:50.894438 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:50.878914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41093 (local address 127.21.43.66:52081)
0810 14:35:50.879627 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:50.879699 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:50.881392 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:50.882007 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:50.882021 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:50.882593 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:50.883391 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:50.883421 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.884604 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.884607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:50.885547 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:50.885557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:50.886107 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:50.886112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:50.886347 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:50.887047 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:50.887065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:50.888794 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:50.890767 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:50.890773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:50.890793 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:50.891088 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:50.891444 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:50.891448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:50.891450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:50.891607 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:50.892093 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:50.892102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:50.892417 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 14:35:50.892634 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 14:35:50.893064 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":219,"threads_started":1}
I20260810 14:35:50.896009 28172 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41093
I20260810 14:35:50.896361 28172 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:50.897003 28172 heartbeater.cc:511] Master 127.21.43.126:41093 requested a full tablet report, sending...
I20260810 14:35:50.898272 27817 ts_manager.cc:194] Registered new tserver with Master: 655a8d88443244a698d72dd2e0741960 (127.21.43.66:38285)
I20260810 14:35:50.899295 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:52081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:35:50.912384 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:51.033210 28179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:51.033504 28179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:51.033550 28179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:51.037779 28179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:51.037860 28179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:51.037961 28179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:51.043572 28179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:51.044852 28179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:51.046480 28179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:51.057261 28188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.057279 28186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.057318 28185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:51.057821 28179 server_base.cc:1034] running on GCE node
I20260810 14:35:51.058269 28179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:51.058959 28179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:51.060185 28179 hybrid_clock.cc:648] HybridClock initialized: now 1786372551060145 us; error 54 us; skew 500 ppm
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.063366 28179 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:35:51.064426 28179 webserver.cc:460] Webserver started at http://127.21.43.67:35969/ using document root <none> and password file <none>
I20260810 14:35:51.065024 28179 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:51.065091 28179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:51.065339 28179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:51.067636 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "bcd591bb5e2b4fdc8a57986e1a7414ab"
format_stamp: "Formatted at 2026-08-10 14:35:51 on dist-test-slave-chsk"
server_key: "990c4a7164bcf5ce19e9cd1376140e5a"
server_key_iv: "db4d5b1fe6cd456b7085ea361fcf99d3"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.068255 28179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "bcd591bb5e2b4fdc8a57986e1a7414ab"
format_stamp: "Formatted at 2026-08-10 14:35:51 on dist-test-slave-chsk"
server_key: "990c4a7164bcf5ce19e9cd1376140e5a"
server_key_iv: "db4d5b1fe6cd456b7085ea361fcf99d3"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.072613 28179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:35:51.076645 28197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:51.078636 28179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:51.078809 28179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "bcd591bb5e2b4fdc8a57986e1a7414ab"
format_stamp: "Formatted at 2026-08-10 14:35:51 on dist-test-slave-chsk"
server_key: "990c4a7164bcf5ce19e9cd1376140e5a"
server_key_iv: "db4d5b1fe6cd456b7085ea361fcf99d3"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.078985 28179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:51.092576 28179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:51.096936 28179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:51.097206 28179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:51.097934 28179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:51.098928 28179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:51.099017 28179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:51.099125 28179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:51.099192 28179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 14:35:51.109417 28179 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:35319
I20260810 14:35:51.109462 28318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:35319 every 8 connection(s)
I20260810 14:35:51.110455 28179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:51.110965 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28179
I20260810 14:35:51.111183 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:51.111723 21677 external_mini_cluster.cc:1428] Setting key b326605b4e96dfe433c3e7395c3e2470
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:35:51.128886 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.113549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:53279 (local address 127.21.43.126:41093)
0810 14:35:51.113701 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.113705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.115059 (+ 1354us) server_negotiation.cc:407] Connection header received
0810 14:35:51.116646 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.116649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.116706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.116815 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.118123 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.118660 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.119979 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.120208 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.120328 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:51.123938 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:51.123942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:51.123945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:51.123984 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:51.126093 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.126900 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.126904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.126905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.126991 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.127633 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.127636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.127638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.127856 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:51.127972 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:51.128552 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 14:35:51.128708 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:35:51.129961 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.114244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41093 (local address 127.21.43.67:53279)
0810 14:35:51.114878 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:51.114963 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:51.116397 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:51.117010 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:51.117018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:51.117444 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:51.117958 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:51.117973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.118796 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.118800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:51.119832 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:51.119843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.120379 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.120383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:51.120565 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:51.121323 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:51.121350 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:51.123598 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:51.126250 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:51.126260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:51.126286 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:51.126780 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:51.127259 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:51.127262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:51.127263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:51.127512 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:51.127972 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:51.127981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:51.128288 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 14:35:51.128494 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:35:51.128875 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":189,"threads_started":1}
I20260810 14:35:51.131786 28319 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41093
I20260810 14:35:51.132157 28319 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:51.132790 28319 heartbeater.cc:511] Master 127.21.43.126:41093 requested a full tablet report, sending...
I20260810 14:35:51.134207 27817 ts_manager.cc:194] Registered new tserver with Master: bcd591bb5e2b4fdc8a57986e1a7414ab (127.21.43.67:35319)
I20260810 14:35:51.134876 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:53279
I20260810 14:35:51.139974 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/test-admin.keytab.
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.161580 21677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 14:35:51.164836 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:51.163049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.21.43.65:38477)
0810 14:35:51.163419 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.163425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.163451 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:35:51.163502 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.163507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.163708 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.163834 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.164620 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33828: BlockingRecv error: recv got EOF from 127.0.0.1:33828 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":147,"threads_started":1}
W20260810 14:35:51.164841 21677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.165990 21677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.167507 21677 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:51.178731 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.168601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.21.43.65:38477)
0810 14:35:51.168739 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.168745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.168836 (+ 91us) server_negotiation.cc:407] Connection header received
0810 14:35:51.169002 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.169005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.169058 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.169145 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.169724 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.170685 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.171612 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.172071 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.172189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:51.174864 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:51.174875 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:51.174891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:51.174917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:51.176811 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.177309 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.177315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.177316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.177401 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.177696 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.177700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.177701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.177991 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:51.178227 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:51.178411 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 14:35:51.178573 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 14:35:51.183169 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:51.181970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.21.43.65:38477)
0810 14:35:51.182134 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.182138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.182238 (+ 100us) server_negotiation.cc:407] Connection header received
0810 14:35:51.182391 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.182394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.182437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.182525 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.183023 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33850: BlockingRecv error: recv got EOF from 127.0.0.1:33850 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 14:35:51.183565 21677 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc)
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.184734 21677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.185760 21677 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:51.195591 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.186759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.21.43.65:38477)
0810 14:35:51.186909 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.186913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.187018 (+ 105us) server_negotiation.cc:407] Connection header received
0810 14:35:51.187206 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.187209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.187262 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.187348 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.187880 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.188393 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.189230 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.189460 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.189574 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:51.192134 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:51.192137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:51.192139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:51.192165 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:51.193709 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.194211 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.194214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.194215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.194304 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.194683 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.194691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.194693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.194882 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:51.195055 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:51.195298 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:35:51.195422 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 14:35:51.199222 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:51.197871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.21.43.65:38477)
0810 14:35:51.198143 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.198147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.198331 (+ 184us) server_negotiation.cc:407] Connection header received
0810 14:35:51.198404 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.198406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.198447 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.198518 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.199062 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33866: BlockingRecv error: recv got EOF from 127.0.0.1:33866 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260810 14:35:51.199306 21677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372517778458-21677-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.200661 21677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.201710 21677 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[27769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:35:51.211252 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.202893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.21.43.65:38477)
0810 14:35:51.203028 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:51.203032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:51.203067 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:35:51.203273 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:51.203275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:51.203319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:51.203389 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:51.203894 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.204399 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.205210 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:51.205416 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:51.205495 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:51.208144 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:51.208148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:51.208150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:51.208175 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:51.209546 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.209984 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.209987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.209988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.210084 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:51.210346 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:51.210348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:51.210349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:51.210524 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:51.210648 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:51.210836 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:35:51.210975 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:35:51.212453 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27885
I20260810 14:35:51.222390 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28032
I20260810 14:35:51.232913 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28179
I20260810 14:35:51.242586 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 27783
2026-08-10T14:35:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2391 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[28341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[28341](info): set up 2 sockets
Aug 10 14:35:51 dist-test-slave-chsk krb5kdc[28341](info): commencing operation
krb5kdc: starting...
W20260810 14:35:52.434619 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:52 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab.
Entry for principal oryx/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab.
I20260810 14:35:52.576665 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34473
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab
--principal=oryx/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:52.692983 28354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:52.693306 28354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:52.693390 28354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:52.697532 28354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:52.697633 28354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:52.697674 28354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:52.697731 28354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:52.697773 28354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:52.703105 28354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab
--principal=oryx/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:52.704205 28354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:52.705173 28354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:52.713747 28359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.713821 28362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.713738 28360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:52.714237 28354 server_base.cc:1034] running on GCE node
I20260810 14:35:52.714875 28354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:52.716409 28354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:52.717630 28354 hybrid_clock.cc:648] HybridClock initialized: now 1786372552717599 us; error 51 us; skew 500 ppm
Aug 10 14:35:52 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:52.720528 28354 init.cc:377] Logged in from keytab as oryx/127.21.43.126@KRBTEST.COM (short username oryx)
I20260810 14:35:52.721510 28354 webserver.cc:460] Webserver started at http://127.21.43.126:43623/ using document root <none> and password file <none>
I20260810 14:35:52.722114 28354 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:52.722224 28354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:52.722462 28354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:52.724699 28354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "a11b119fca8349e8920a016bf590fd25"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "bed1c9c2d5bfbcfa911ba4c1867e24a2"
server_key_iv: "98cce73043d5f3cc2bd21adadb5013e7"
server_key_version: "encryptionkey@0"
I20260810 14:35:52.725337 28354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "a11b119fca8349e8920a016bf590fd25"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "bed1c9c2d5bfbcfa911ba4c1867e24a2"
server_key_iv: "98cce73043d5f3cc2bd21adadb5013e7"
server_key_version: "encryptionkey@0"
I20260810 14:35:52.729354 28354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:35:52.731907 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.733119 28354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:35:52.733284 28354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "a11b119fca8349e8920a016bf590fd25"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "bed1c9c2d5bfbcfa911ba4c1867e24a2"
server_key_iv: "98cce73043d5f3cc2bd21adadb5013e7"
server_key_version: "encryptionkey@0"
I20260810 14:35:52.733407 28354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:52.750212 28354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:52.753392 28354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:52.753610 28354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:52.761109 28354 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:34473
I20260810 14:35:52.761433 28426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:34473 every 8 connection(s)
I20260810 14:35:52.762168 28354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:52.764469 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28354
I20260810 14:35:52.764595 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:52.764919 21677 external_mini_cluster.cc:1428] Setting key 94fbe3e8ff9596d0bb318eebac540e88
I20260810 14:35:52.765327 28427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:52.772783 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Bootstrap starting.
Aug 10 14:35:52 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:52.775470 28427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:52.776494 28427 log.cc:824] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:52.778844 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: No bootstrap required, opened a new log
I20260810 14:35:52.780520 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.766484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.21.43.126:34473)
0810 14:35:52.766932 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:52.766951 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:52.766995 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:35:52.767795 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:52.767819 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:52.768236 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:52.768621 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:52.769344 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.770877 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.771863 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.772518 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.772692 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:52.775424 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:52.775431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:52.775448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:52.775484 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:52.777357 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.777789 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.777795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.777800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.777938 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.778221 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.778224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.778226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.778744 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:52.778939 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:52.779290 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 14:35:52.779605 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260810 14:35:52.783329 28427 raft_consensus.cc:348] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } }
I20260810 14:35:52.783550 28427 raft_consensus.cc:374] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:52.783587 28427 raft_consensus.cc:729] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11b119fca8349e8920a016bf590fd25, State: Initialized, Role: FOLLOWER
I20260810 14:35:52.783938 28427 consensus_queue.cc:260] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } }
I20260810 14:35:52.784068 28427 raft_consensus.cc:388] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:52.784109 28427 raft_consensus.cc:482] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:52.784180 28427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:52.785066 28427 raft_consensus.cc:504] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } }
I20260810 14:35:52.785360 28427 leader_election.cc:302] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a11b119fca8349e8920a016bf590fd25; no voters:
I20260810 14:35:52.785641 28427 leader_election.cc:288] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:52.785859 28432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:52.786141 28432 raft_consensus.cc:686] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 LEADER]: Becoming Leader. State: Replica: a11b119fca8349e8920a016bf590fd25, State: Running, Role: LEADER
I20260810 14:35:52.786541 28432 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } }
I20260810 14:35:52.786983 28427 sys_catalog.cc:564] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:52.788295 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11b119fca8349e8920a016bf590fd25. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } }
I20260810 14:35:52.788444 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:52.788872 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } }
I20260810 14:35:52.788954 28442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:52.789923 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:52.792598 28442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:52.799608 28442 catalog_manager.cc:1347] Generated new cluster ID: 8fa5849c604f456b97318e724a298f72
I20260810 14:35:52.799705 28442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:52.804836 28442 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:52.806059 28442 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:52.811024 28442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Generated new TSK 0
I20260810 14:35:52.811889 28442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
I20260810 14:35:52.846464 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:52.965909 28456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:52.966188 28456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:52.966240 28456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:52.970332 28456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:52.970402 28456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:52.970490 28456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:52.976148 28456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:52.977319 28456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:52.978370 28456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:52.988025 28461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.987998 28464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.988067 28462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:52.988273 28456 server_base.cc:1034] running on GCE node
I20260810 14:35:52.988860 28456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:52.989530 28456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:52.990746 28456 hybrid_clock.cc:648] HybridClock initialized: now 1786372552990730 us; error 28 us; skew 500 ppm
Aug 10 14:35:52 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:52.993820 28456 init.cc:377] Logged in from keytab as oryx/127.21.43.65@KRBTEST.COM (short username oryx)
I20260810 14:35:52.994767 28456 webserver.cc:460] Webserver started at http://127.21.43.65:37889/ using document root <none> and password file <none>
I20260810 14:35:52.995429 28456 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:52.995525 28456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:52.995774 28456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:52.997912 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "8efb7e93937744829323a54d7d7d1f20"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "4d7cd26d6b6a9691aee4b8fe4c135157"
server_key_iv: "cd26d220bb9bf3e336c2752ede1b4738"
server_key_version: "encryptionkey@0"
I20260810 14:35:52.998520 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "8efb7e93937744829323a54d7d7d1f20"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "4d7cd26d6b6a9691aee4b8fe4c135157"
server_key_iv: "cd26d220bb9bf3e336c2752ede1b4738"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.002480 28456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:53.005069 28472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.006502 28456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:35:53.006637 28456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "8efb7e93937744829323a54d7d7d1f20"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "4d7cd26d6b6a9691aee4b8fe4c135157"
server_key_iv: "cd26d220bb9bf3e336c2752ede1b4738"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.006752 28456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.028352 28456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:53.031895 28456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:53.032145 28456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:53.032831 28456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:53.033756 28456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:53.033807 28456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.033849 28456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:53.033902 28456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.043292 28456 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:42309
I20260810 14:35:53.043330 28592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:42309 every 8 connection(s)
I20260810 14:35:53.044241 28456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:53.044622 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28456
I20260810 14:35:53.044772 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:53.045117 21677 external_mini_cluster.cc:1428] Setting key 6756f8474140bcbb84ce92d466397b7d
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.66@KRBTEST.COM" created.
I20260810 14:35:53.060645 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.046126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:34063 (local address 127.21.43.126:34473)
0810 14:35:53.046259 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.046270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.047373 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 14:35:53.048804 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.048807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.048854 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.048925 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.050459 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.051003 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.052920 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.053172 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.053269 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.056515 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.056521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.056527 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.056568 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.058247 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.058848 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.058850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.058852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.058928 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.059348 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.059351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.059353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.059558 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.059639 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.060303 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.060447 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:35:53.061486 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.046650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:34063)
0810 14:35:53.047209 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.047260 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.048549 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.049113 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.049123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.049675 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.050301 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.050317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.051186 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.051193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.052753 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.052768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.053361 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.053368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.053611 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.054348 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.054371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.056297 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.058419 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.058428 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.058449 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.058743 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.059007 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.059009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.059011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.059224 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.059655 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.059661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.059989 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.060237 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.060591 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260810 14:35:53.062881 28593 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:53.063223 28593 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:53.063786 28593 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:53.065495 28388 ts_manager.cc:194] Registered new tserver with Master: 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309)
I20260810 14:35:53.066705 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:34063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
I20260810 14:35:53.078631 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:53.195551 28598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:53.195822 28598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:53.195865 28598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:53.199963 28598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:53.200029 28598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:53.200088 28598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:53.205371 28598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:53.206507 28598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:53.207592 28598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:53.217883 28604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.217908 28598 server_base.cc:1034] running on GCE node
W20260810 14:35:53.217854 28606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.217854 28603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.218544 28598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:53.219231 28598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:53.220431 28598 hybrid_clock.cc:648] HybridClock initialized: now 1786372553220404 us; error 54 us; skew 500 ppm
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:53.223331 28598 init.cc:377] Logged in from keytab as oryx/127.21.43.66@KRBTEST.COM (short username oryx)
I20260810 14:35:53.224207 28598 webserver.cc:460] Webserver started at http://127.21.43.66:45597/ using document root <none> and password file <none>
I20260810 14:35:53.224779 28598 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:53.224870 28598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:53.225099 28598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:53.227240 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "936640a192a6d88fe926f7542a3488cf"
server_key_iv: "537c83057bef4d7518673a94ba2a73d9"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.227803 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "936640a192a6d88fe926f7542a3488cf"
server_key_iv: "537c83057bef4d7518673a94ba2a73d9"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.231510 28598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 14:35:53.233954 28614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.235221 28598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:35:53.235363 28598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "936640a192a6d88fe926f7542a3488cf"
server_key_iv: "537c83057bef4d7518673a94ba2a73d9"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.235474 28598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.252381 28598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:53.255565 28598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:53.255784 28598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:53.256421 28598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:53.257346 28598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:53.257417 28598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.257488 28598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:53.257537 28598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.267505 28598 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34111
I20260810 14:35:53.267536 28736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34111 every 8 connection(s)
I20260810 14:35:53.268460 28598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:53.277501 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28598
I20260810 14:35:53.277645 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:53.277915 21677 external_mini_cluster.cc:1428] Setting key b94c6a8bb88cf2a5c30cdd7e001ea2e5
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:53.283371 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.270376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:53795 (local address 127.21.43.126:34473)
0810 14:35:53.270571 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.270575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.271471 (+ 896us) server_negotiation.cc:407] Connection header received
0810 14:35:53.272825 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.272828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.272888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.272995 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.274171 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.274707 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.276054 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.276316 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.276399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.279264 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.279267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.279270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.279297 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.281108 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.281622 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.281628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.281631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.281743 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.282194 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.282198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.282199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.282375 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.282448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.283091 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.283250 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:35:53.284575 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.270782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:53795)
0810 14:35:53.271317 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.271362 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.272610 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.273186 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.273193 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.273554 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.274006 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.274018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.274846 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.274856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.275877 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.275887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.276464 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.276467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.276608 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.277248 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.277266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.279073 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.281233 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.281239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.281251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.281505 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.281852 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.281858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.281861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.282091 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.282481 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.282489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.282827 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.283057 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.283452 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260810 14:35:53.285799 28737 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:53.286060 28737 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:53.286547 28737 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:53.287703 28388 ts_manager.cc:194] Registered new tserver with Master: 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:53.288456 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:53795
WARNING: no policy specified for oryx/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
I20260810 14:35:53.307605 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:53.420372 28742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:53.420776 28742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:53.420893 28742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:53.425316 28742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:53.425400 28742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:53.425493 28742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:53.430819 28742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:53.431985 28742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:53.433037 28742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:53.442823 28747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.442873 28750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.442823 28748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.443068 28742 server_base.cc:1034] running on GCE node
I20260810 14:35:53.443562 28742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:53.444260 28742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:53.445433 28742 hybrid_clock.cc:648] HybridClock initialized: now 1786372553445414 us; error 30 us; skew 500 ppm
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:53.448302 28742 init.cc:377] Logged in from keytab as oryx/127.21.43.67@KRBTEST.COM (short username oryx)
I20260810 14:35:53.449123 28742 webserver.cc:460] Webserver started at http://127.21.43.67:40293/ using document root <none> and password file <none>
I20260810 14:35:53.449652 28742 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:53.449713 28742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:53.449880 28742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:53.451933 28742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "26fabb5275c345c78f1e0d469de75525"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "131b0775140bc41f25ce99765f1feef3"
server_key_iv: "50dde72f716a4aef98254313dbaf7086"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.452443 28742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "26fabb5275c345c78f1e0d469de75525"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "131b0775140bc41f25ce99765f1feef3"
server_key_iv: "50dde72f716a4aef98254313dbaf7086"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.456180 28742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:53.458621 28757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.459928 28742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:35:53.460031 28742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "26fabb5275c345c78f1e0d469de75525"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "131b0775140bc41f25ce99765f1feef3"
server_key_iv: "50dde72f716a4aef98254313dbaf7086"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.460125 28742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.472978 28742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:53.476715 28742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:53.476886 28742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:53.477442 28742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:53.478253 28742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:53.478302 28742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.478343 28742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:53.478358 28742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.487572 28742 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:38713
I20260810 14:35:53.487614 28885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:38713 every 8 connection(s)
I20260810 14:35:53.488575 28742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:53.494817 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28742
I20260810 14:35:53.494972 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:53.495332 21677 external_mini_cluster.cc:1428] Setting key 39312d5f3e21ee350fe4b35c7535c4d9
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:53.502645 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.490640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:36323 (local address 127.21.43.126:34473)
0810 14:35:53.490820 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.490826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.491687 (+ 861us) server_negotiation.cc:407] Connection header received
0810 14:35:53.492976 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.492979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.493035 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.493110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.494315 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.494835 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.496168 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.496390 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.496505 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.499011 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.499015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.499022 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.499047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.500484 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.501036 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.501040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.501043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.501130 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.501490 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.501494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.501496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.501700 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.501810 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.502332 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.502471 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:35:53.503603 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.491037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:36323)
0810 14:35:53.491519 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.491561 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.492787 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.493272 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.493279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.493641 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.494110 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.494149 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.494994 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.494998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.496036 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.496046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.496553 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.496558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.496728 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.497369 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.497386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.498838 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.500623 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.500630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.500647 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.500924 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.501232 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.501235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.501237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.501387 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.501809 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.501816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.502084 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.502290 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.502630 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260810 14:35:53.504733 28886 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:53.504997 28886 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:53.505535 28886 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:53.506620 28388 ts_manager.cc:194] Registered new tserver with Master: 26fabb5275c345c78f1e0d469de75525 (127.21.43.67:38713)
I20260810 14:35:53.507269 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:36323
I20260810 14:35:53.509387 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.43.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.125.keytab.
Entry for principal oryx/127.21.43.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.125.keytab.
I20260810 14:35:53.541872 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:41739
--webserver_interface=127.21.43.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.125.keytab
--principal=oryx/127.21.43.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34473,127.21.43.125:41739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:53.659179 28892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:53.659466 28892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:53.659518 28892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:53.663645 28892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:53.663708 28892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:53.663727 28892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:53.663744 28892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:53.663759 28892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:53.669332 28892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34473,127.21.43.125:41739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.125.keytab
--principal=oryx/127.21.43.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:41739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.43.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
W20260810 14:35:53.669545 28892 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.43.126:34473,127.21.43.125:41739'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 14:35:53.670441 28892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:53.671515 28892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:53.680526 28899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.680526 28901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.680522 28898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.680757 28892 server_base.cc:1034] running on GCE node
I20260810 14:35:53.681586 28892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:53.682786 28892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:53.683983 28892 hybrid_clock.cc:648] HybridClock initialized: now 1786372553683936 us; error 64 us; skew 500 ppm
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:53.686905 28892 init.cc:377] Logged in from keytab as oryx/127.21.43.125@KRBTEST.COM (short username oryx)
I20260810 14:35:53.687922 28892 webserver.cc:460] Webserver started at http://127.21.43.125:36785/ using document root <none> and password file <none>
I20260810 14:35:53.688622 28892 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:53.688719 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:53.688969 28892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:53.691126 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/instance:
uuid: "50c91f3147b84f338f17ef174822d6a2"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "db4b1d5117a594fad4512a7aea114094"
server_key_iv: "302747dd24be0a3a2c73e4ffdfebae10"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.691846 28892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal/instance:
uuid: "50c91f3147b84f338f17ef174822d6a2"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "db4b1d5117a594fad4512a7aea114094"
server_key_iv: "302747dd24be0a3a2c73e4ffdfebae10"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.695785 28892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:35:53.698271 28910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.699753 28892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:35:53.699903 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "50c91f3147b84f338f17ef174822d6a2"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "db4b1d5117a594fad4512a7aea114094"
server_key_iv: "302747dd24be0a3a2c73e4ffdfebae10"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.700022 28892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.712366 28892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:53.715567 28892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:53.715776 28892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:53.723457 28892 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.125:41739
I20260810 14:35:53.723488 28965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.125:41739 every 8 connection(s)
I20260810 14:35:53.724485 28892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
I20260810 14:35:53.727521 28966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:53.729743 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 28892
I20260810 14:35:53.732486 28966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.125@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:53.747249 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.734835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.21.43.126:34473)
0810 14:35:53.734974 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.734979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.735790 (+ 811us) server_negotiation.cc:407] Connection header received
0810 14:35:53.735916 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.735921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.735985 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.736083 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.737185 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.738095 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.739620 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.739991 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.740092 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.742821 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.742825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.742828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.742864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.744669 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.745298 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.745304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.745309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.745408 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.745829 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.745833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.745834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.746150 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.746255 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.746896 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.747033 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:53.748140 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.734765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49108)
0810 14:35:53.735337 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.735394 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.735526 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.736259 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.736265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.736762 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.737021 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.737038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.738273 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.738277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.739495 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.739505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.740268 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.740271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.740356 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.740999 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.741015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.742650 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.744809 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.744818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.744827 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.745182 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.745504 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.745507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.745509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.745710 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.746261 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.746267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.746572 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.746811 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.747219 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"spinlock_wait_cycles":896,"thread_start_us":103,"threads_started":1}
I20260810 14:35:53.748148 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.731758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.21.43.125:41739)
0810 14:35:53.732416 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.732467 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.733211 (+ 744us) server_negotiation.cc:407] Connection header received
0810 14:35:53.734168 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.734191 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.734607 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.734951 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.736407 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.737961 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.739336 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.739832 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.740086 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.743185 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.743193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.743216 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.743243 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.745215 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.745748 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.745752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.745754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.745864 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.746150 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.746155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.746156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.746526 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.746672 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.746975 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.747368 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":198,"server-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260810 14:35:53.749907 28966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.125@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:53.758492 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.750543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42214)
0810 14:35:53.750674 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.750690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.750770 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.751048 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.751050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.751261 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.751360 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.751365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.752423 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.752426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.752931 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.752938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.753374 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.753378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.753460 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.754005 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.754017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.755370 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.756994 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.756997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.756998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.757297 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.757568 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.757570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.757572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.757694 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.758091 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.758093 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.758173 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.758255 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.758363 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:35:53.758656 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.750734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.21.43.125:41739)
0810 14:35:53.750857 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.750861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.750872 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:35:53.750900 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.750902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.750933 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.750995 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.751457 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.752318 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.753025 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.753249 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.753324 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.755509 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.755512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.755513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.755534 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.756876 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.757399 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.757402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.757403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.757481 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.757803 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.757806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.757807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.757987 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.758069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.758352 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.758455 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:53.761884 28966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Bootstrap starting.
I20260810 14:35:53.764761 28966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:53.765617 28966 log.cc:824] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:53.767912 28966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: No bootstrap required, opened a new log
I20260810 14:35:53.772099 28966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } }
I20260810 14:35:53.772351 28966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:53.772405 28966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c91f3147b84f338f17ef174822d6a2, State: Initialized, Role: FOLLOWER
I20260810 14:35:53.772881 28966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } }
I20260810 14:35:53.773420 28972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } } }
I20260810 14:35:53.773563 28972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:53.774273 28966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:35:53.779755 28985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:35:53.779857 28985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:35:53.781832 28892 master_runner.cc:428] Detected that this master 50c91f3147b84f338f17ef174822d6a2 is joining an existing cluster
I20260810 14:35:53.781903 28892 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.43.125:41739
I20260810 14:35:53.793236 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.784359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42218)
0810 14:35:53.785082 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.785096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.785228 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.785485 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.785487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.785717 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.785815 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.785820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.786671 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.786675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.787702 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.787713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.788205 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.788208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.788298 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.788878 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.788889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.789564 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.791600 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.791604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.791606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.791821 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.792094 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.792098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.792100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.792205 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.792606 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.792610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.792724 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.792862 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.793015 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":275,"spinlock_wait_cycles":5504,"thread_start_us":99,"threads_started":1}
I20260810 14:35:53.793725 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.784486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.21.43.125:41739)
0810 14:35:53.784601 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.784606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.785164 (+ 558us) server_negotiation.cc:407] Connection header received
0810 14:35:53.785333 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.785338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.785384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.785468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.785925 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.786551 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.787843 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.788075 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.788184 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.789702 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.789705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.789708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.789729 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.791473 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.791920 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.791925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.791927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.792005 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.792308 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.792313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.792317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.792503 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.792579 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.792913 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.793562 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":507,"server-negotiator.queue_time_us":49}
I20260810 14:35:53.794420 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.784359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49114)
0810 14:35:53.784949 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.784965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.785148 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.785798 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.785801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.786029 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.786163 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.786171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.787032 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.787037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.787786 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.787800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.788408 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.788413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.788563 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.789059 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.789071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.789564 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.792003 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.792006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.792008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.792221 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.792809 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.792813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.792815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.792900 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.793634 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.793637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.793744 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.794094 (+ 350us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.794251 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":126,"threads_started":1}
I20260810 14:35:53.794415 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.784449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.21.43.126:34473)
0810 14:35:53.784600 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.784606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.785065 (+ 459us) server_negotiation.cc:407] Connection header received
0810 14:35:53.785515 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.785518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.785586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.785644 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.786295 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.786921 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.787910 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.788285 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.788399 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.789720 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.789725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.789728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.789760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.791876 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.792623 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.792625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.792627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.792709 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.793021 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.793024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.793026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.793214 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.793865 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.794151 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.794274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.126@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:53.810843 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.799446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.21.43.125:41739)
0810 14:35:53.799570 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.799575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.799934 (+ 359us) server_negotiation.cc:407] Connection header received
0810 14:35:53.800099 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.800102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.800146 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.800218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.801027 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.801625 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.802961 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.803320 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.803393 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.806615 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.806620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.806622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.806651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.808457 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.809121 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.809125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.809127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.809222 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.809620 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.809623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.809625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.809863 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.809967 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.810418 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.810673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:53.811066 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.799375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42230)
0810 14:35:53.799782 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.799843 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.799983 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.800272 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.800278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.800758 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.800899 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.800906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.801740 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.801743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.802825 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.802837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.803563 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.803567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.803744 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.804604 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.804620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.806441 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.808592 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.808605 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.808614 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.808999 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.809338 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.809344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.809348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.809463 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.809994 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.810001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.810159 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.810349 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.810661 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for oryx/127.21.43.124@KRBTEST.COM; defaulting to no policy
I20260810 14:35:53.812865 28388 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.43.125:41739: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50c91f3147b84f338f17ef174822d6a2"
member_type: NON_VOTER
last_known_addr {
host: "127.21.43.125"
port: 41739
}
attrs {
promote: true
}
}
dest_uuid: "a11b119fca8349e8920a016bf590fd25"
cas_config_opid_index: -1
I20260810 14:35:53.813714 28388 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.124@KRBTEST.COM" created.
I20260810 14:35:53.816164 28997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } }
I20260810 14:35:53.816996 28939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:53.818385 28997 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.43.125:41739
I20260810 14:35:53.818571 28939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 14:35:53.818683 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } }
I20260810 14:35:53.818797 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:53.819171 28892 master.cc:537] Master@127.21.43.125:41739 shutting down...
I20260810 14:35:53.819947 28997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:35:53.822373 28374 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 14:35:53.823660 28374 consensus_peers.cc:487] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 -> Peer 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125:41739): Couldn't send request to peer 50c91f3147b84f338f17ef174822d6a2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:35:53.824407 28892 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:53.824769 28892 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:53.824887 28892 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.124.keytab.
Entry for principal oryx/127.21.43.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.124.keytab.
I20260810 14:35:53.836520 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:46269
--webserver_interface=127.21.43.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.124.keytab
--principal=oryx/127.21.43.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:53.840598 28892 master.cc:552] Master@127.21.43.125:41739 shutdown complete.
I20260810 14:35:53.840668 28892 master_runner.cc:305] Clearing existing system tablet
I20260810 14:35:53.840922 28892 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:53.840981 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:53.842402 28892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:35:53.843431 29004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.843711 28892 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.843983 28892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:35:53.844067 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "50c91f3147b84f338f17ef174822d6a2"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "db4b1d5117a594fad4512a7aea114094"
server_key_iv: "302747dd24be0a3a2c73e4ffdfebae10"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.846441 28892 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:53.849144 28892 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:35:53.849324 28892 log.cc:1196] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 14:35:53.849725 28892 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Deleting consensus metadata
I20260810 14:35:53.849963 28892 master_runner.cc:315] Copying system tablet from 127.21.43.126:34473
I20260810 14:35:53.851437 28892 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: tablet copy: Beginning tablet copy session from remote peer at address 127.21.43.126:34473
I20260810 14:35:53.861128 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.852307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.21.43.126:34473)
0810 14:35:53.852452 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:53.852456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:53.852633 (+ 177us) server_negotiation.cc:407] Connection header received
0810 14:35:53.852752 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:53.852756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:53.852812 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:53.852918 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:53.853466 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.854286 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.855567 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.855919 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.856031 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.857428 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:53.857432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:53.857435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:53.857466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:53.859233 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.859760 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.859764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.859766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.859860 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:53.860177 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:53.860181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:53.860182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:53.860417 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:53.860519 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:53.860835 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 14:35:53.860966 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:53.861127 29011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.852182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49128)
0810 14:35:53.852525 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.852541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:53.852665 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:53.852950 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:53.852953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:53.853185 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:53.853329 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.853337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.854423 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.854427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.855429 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:53.855442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:53.856051 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:53.856055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:53.856178 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:53.856795 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:53.856810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:53.857263 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:53.859367 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.859370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.859372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.859652 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.859962 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:53.859965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:53.859967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:53.860076 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:53.860530 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:53.860536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:53.860647 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:35:53.860756 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 14:35:53.860912 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20260810 14:35:53.862237 28410 tablet_copy_service.cc:140] P a11b119fca8349e8920a016bf590fd25: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50c91f3147b84f338f17ef174822d6a2 ({username='oryx', principal='oryx/127.21.43.125@KRBTEST.COM'} at 127.0.0.1:49128)
I20260810 14:35:53.862445 28410 tablet_copy_service.cc:161] P a11b119fca8349e8920a016bf590fd25: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50c91f3147b84f338f17ef174822d6a2 at {username='oryx', principal='oryx/127.21.43.125@KRBTEST.COM'} at 127.0.0.1:49128: session id = 50c91f3147b84f338f17ef174822d6a2-00000000000000000000000000000000
I20260810 14:35:53.864373 28410 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:35:53.866323 28892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:35:53.869067 28892 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: tablet copy: Starting download of 0 data blocks...
I20260810 14:35:53.869285 28892 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: tablet copy: Starting download of 1 WAL segments...
I20260810 14:35:53.871446 28892 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:35:53.873530 28410 tablet_copy_service.cc:342] P a11b119fca8349e8920a016bf590fd25: Request end of tablet copy session 50c91f3147b84f338f17ef174822d6a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.43.125@KRBTEST.COM'} at 127.0.0.1:49128
I20260810 14:35:53.873679 28410 tablet_copy_service.cc:433] P a11b119fca8349e8920a016bf590fd25: ending tablet copy session 50c91f3147b84f338f17ef174822d6a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50c91f3147b84f338f17ef174822d6a2
I20260810 14:35:53.882441 28892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:53.884372 29012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:53.888293 29013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.888813 28892 server_base.cc:1034] running on GCE node
W20260810 14:35:53.889209 29016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:53.889526 28892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:53.889986 28892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:53.891264 28892 hybrid_clock.cc:648] HybridClock initialized: now 1786372553891251 us; error 31 us; skew 500 ppm
Aug 10 14:35:53 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372553, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:53.892979 28892 init.cc:377] Logged in from keytab as oryx/127.21.43.125@KRBTEST.COM (short username oryx)
I20260810 14:35:53.893527 28892 webserver.cc:460] Webserver started at http://127.21.43.125:43199/ using document root <none> and password file <none>
I20260810 14:35:53.893743 28892 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:53.893785 28892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:53.895293 28892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:53.896382 29024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:53.896646 28892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:53.896764 28892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "50c91f3147b84f338f17ef174822d6a2"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "db4b1d5117a594fad4512a7aea114094"
server_key_iv: "302747dd24be0a3a2c73e4ffdfebae10"
server_key_version: "encryptionkey@0"
I20260810 14:35:53.896914 28892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:53.914300 28892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:53.917135 28892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:53.923225 28892 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.125:41739
I20260810 14:35:53.923242 29077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.125:41739 every 8 connection(s)
I20260810 14:35:53.923879 28892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
I20260810 14:35:53.925423 29078 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:35:53.926323 29078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Bootstrap starting.
I20260810 14:35:53.935742 29078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:35:53.936182 29078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: Bootstrap complete.
I20260810 14:35:53.936872 29078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } }
I20260810 14:35:53.937057 29078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50c91f3147b84f338f17ef174822d6a2, State: Initialized, Role: LEARNER
I20260810 14:35:53.937183 29078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } }
I20260810 14:35:53.937638 29082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } }
I20260810 14:35:53.937808 29082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:35:53.937816 29078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:53.941359 29093 catalog_manager.cc:1259] Loaded cluster ID: 8fa5849c604f456b97318e724a298f72
I20260810 14:35:53.941418 29093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: loading cluster ID for follower catalog manager: success
I20260810 14:35:53.943399 29093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: acquiring CA information for follower catalog manager: success
I20260810 14:35:53.944583 29093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:35:53.987442 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:53.987840 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:53.987954 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:53.993496 29002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:53.993587 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:53.993606 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:53.993624 29002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:53.993640 29002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:53.999678 29002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.124.keytab
--principal=oryx/127.21.43.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.43.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:54.000903 29002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:54.002038 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:54.011183 29101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.011235 29099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.011350 29002 server_base.cc:1034] running on GCE node
W20260810 14:35:54.011576 29098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.012164 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:54.013394 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:54.014640 29002 hybrid_clock.cc:648] HybridClock initialized: now 1786372554014608 us; error 34 us; skew 500 ppm
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:54.018036 29002 init.cc:377] Logged in from keytab as oryx/127.21.43.124@KRBTEST.COM (short username oryx)
I20260810 14:35:54.019505 29002 webserver.cc:460] Webserver started at http://127.21.43.124:44901/ using document root <none> and password file <none>
I20260810 14:35:54.020375 29002 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.020501 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.020792 29002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:54.024088 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/instance:
uuid: "2f1c7c9c1f24452a8515d5951b091680"
format_stamp: "Formatted at 2026-08-10 14:35:54 on dist-test-slave-chsk"
server_key: "93f9f9bb1a802ba2a62f294a9b88258b"
server_key_iv: "533bfa793042c3bf848ff84199e78a30"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.024868 29002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal/instance:
uuid: "2f1c7c9c1f24452a8515d5951b091680"
format_stamp: "Formatted at 2026-08-10 14:35:54 on dist-test-slave-chsk"
server_key: "93f9f9bb1a802ba2a62f294a9b88258b"
server_key_iv: "533bfa793042c3bf848ff84199e78a30"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.030736 29002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 14:35:54.034601 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.037184 29002 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 14:35:54.037374 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "2f1c7c9c1f24452a8515d5951b091680"
format_stamp: "Formatted at 2026-08-10 14:35:54 on dist-test-slave-chsk"
server_key: "93f9f9bb1a802ba2a62f294a9b88258b"
server_key_iv: "533bfa793042c3bf848ff84199e78a30"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.037560 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.061473 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:54.065958 29002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:54.066215 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:54.069767 28593 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
I20260810 14:35:54.077996 29002 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.124:46269
I20260810 14:35:54.078409 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.124:46269 every 8 connection(s)
I20260810 14:35:54.079321 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
I20260810 14:35:54.083278 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:54.085579 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29002
I20260810 14:35:54.089660 29166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.124@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:54.103535 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.087450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.21.43.124:46269)
0810 14:35:54.088245 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.088269 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.088320 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:35:54.089310 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.089336 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.089816 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.090233 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.091058 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.092589 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.093741 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.094550 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.094725 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.097592 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.097598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.097607 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.097681 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.099970 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.100559 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.100565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.100572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.100710 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.101071 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.101076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.101079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.101643 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.101878 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.102241 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.102544 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":333,"threads_started":1}
I20260810 14:35:54.105057 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.093652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.21.43.126:34473)
0810 14:35:54.093848 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.093853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.094337 (+ 484us) server_negotiation.cc:407] Connection header received
0810 14:35:54.094476 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.094480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.094540 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.094636 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.095941 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.096480 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.097702 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.098096 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.098231 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.101056 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.101063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.101066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.101097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.103185 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.103710 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.103715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.103719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.103817 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.104131 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.104135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.104137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.104372 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.104478 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.104836 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.104942 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 14:35:54.105150 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.093498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49136)
0810 14:35:54.094083 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.094138 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.094374 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.095128 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.095133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.095661 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.095800 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.095806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.096630 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.096634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.097557 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.097568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.098245 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.098250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.098397 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.099096 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.099114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.100901 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.103291 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.103298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.103303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.103593 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.103931 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.103934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.103936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.104027 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.104481 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.104486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.104618 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.104772 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.104956 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":309,"threads_started":1}
I20260810 14:35:54.106312 29166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.124@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:54.114915 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.106878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42240)
0810 14:35:54.106985 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.107006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.107067 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.107495 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.107498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.107648 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.107747 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.107752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.108630 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.108632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.109174 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.109182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.109646 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.109649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.109746 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.110310 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.110328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.111759 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.113560 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.113562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.113564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.113769 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.114053 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.114055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.114057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.114138 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.114509 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.114511 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.114596 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.114677 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.114787 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 14:35:54.115007 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.106976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.21.43.125:41739)
0810 14:35:54.107267 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.107270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.107283 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:54.107320 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.107322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.107360 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.107438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.107856 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.108509 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.109286 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.109515 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.109598 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.111913 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.111917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.111919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.111944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.113439 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.113865 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.113868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.113869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.113949 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.114233 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.114235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.114236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.114417 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.114493 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.114741 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.114846 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":117,"threads_started":1}
I20260810 14:35:54.115895 29166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.124@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
I20260810 14:35:54.124222 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.116427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.0.0.1:59786)
0810 14:35:54.116576 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.116591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.116678 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.116897 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.116900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.117074 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.117137 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.117142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.118233 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.118237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.118760 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.118767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.119276 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.119279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.119406 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.119856 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.119867 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.121165 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.122725 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.122727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.122729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.122929 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.123292 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.123294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.123295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.123382 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.123744 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.123746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.123829 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.123907 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.124034 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":16}
I20260810 14:35:54.124225 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.116478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.21.43.124:46269)
0810 14:35:54.116576 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.116582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.116636 (+ 54us) server_negotiation.cc:407] Connection header received
0810 14:35:54.116759 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.116764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.116801 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.116886 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.117251 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.117761 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.118872 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.119101 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.119263 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.121307 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.121311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.121313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.121339 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.122611 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.123019 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.123022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.123023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.123093 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.123466 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.123468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.123470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.123650 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.123732 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.123966 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.124091 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":44}
I20260810 14:35:54.127367 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Bootstrap starting.
I20260810 14:35:54.129654 29166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:54.130324 29166 log.cc:824] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:54.132349 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: No bootstrap required, opened a new log
I20260810 14:35:54.135303 29166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } }
I20260810 14:35:54.135500 29166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:54.135535 29166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1c7c9c1f24452a8515d5951b091680, State: Initialized, Role: FOLLOWER
I20260810 14:35:54.135906 29166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } }
I20260810 14:35:54.136340 29172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } } }
I20260810 14:35:54.136541 29172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:54.137217 29166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:35:54.142454 29185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:35:54.142560 29185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:35:54.146119 29002 master_runner.cc:428] Detected that this master 2f1c7c9c1f24452a8515d5951b091680 is joining an existing cluster
I20260810 14:35:54.146191 29002 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.43.124:46269
I20260810 14:35:54.157174 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28456
I20260810 14:35:54.158614 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.148902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.0.0.1:59788)
0810 14:35:54.149417 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.149432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.149531 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.149972 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.149976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.150238 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.150391 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.150398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.151292 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.151295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.152208 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.152219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.152756 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.152850 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.153500 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.153512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.153853 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.155756 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.155763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.155765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.155967 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.156271 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.156274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.156277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.156381 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.156831 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.156834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.156927 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.157045 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.157202 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":179,"threads_started":1}
I20260810 14:35:54.160097 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.149543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.21.43.124:46269)
0810 14:35:54.149645 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.149649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.149662 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:54.149713 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.149717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.149771 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.149854 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.150501 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.151103 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152334 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.152620 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.153985 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.153988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.153991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.154013 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.155646 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.156071 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.156077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.156079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.156157 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.156490 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.156494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.156499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.156722 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.156832 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.157102 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.159980 (+ 2878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":34}
I20260810 14:35:54.160421 29172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:54.160614 29172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } }
I20260810 14:35:54.161561 29172 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473), 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125:41739)
I20260810 14:35:54.162123 29052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f1c7c9c1f24452a8515d5951b091680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c91f3147b84f338f17ef174822d6a2" is_pre_election: true
I20260810 14:35:54.162124 28400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f1c7c9c1f24452a8515d5951b091680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11b119fca8349e8920a016bf590fd25" is_pre_election: true
I20260810 14:35:54.162317 29052 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Handling vote request from an unknown peer 2f1c7c9c1f24452a8515d5951b091680
I20260810 14:35:54.162459 29052 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 2f1c7c9c1f24452a8515d5951b091680 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 14:35:54.162961 29112 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f1c7c9c1f24452a8515d5951b091680; no voters: 50c91f3147b84f338f17ef174822d6a2, a11b119fca8349e8920a016bf590fd25
I20260810 14:35:54.163285 29172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:54.164393 29172 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 14:35:54.175079 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.149067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.21.43.125:41739)
0810 14:35:54.149214 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.149218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.150037 (+ 819us) server_negotiation.cc:407] Connection header received
0810 14:35:54.150230 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.150234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.150288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.150341 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.151046 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.151633 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152501 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.152865 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152985 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.171237 (+ 18252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.171242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.171245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.171279 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.173226 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.173742 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.173746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.173748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.173835 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.174122 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.174125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.174127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.174354 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.174422 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.174786 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.174920 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:35:54.175629 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.148901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42246)
0810 14:35:54.149947 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.149961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.150101 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.150500 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.150503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.150774 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.150908 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.150916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.151750 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.151754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.152376 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.152384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.152989 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.152994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.153128 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.170453 (+ 17325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.170473 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.171023 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.173375 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.173379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.173381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.173620 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.173944 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.173947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.173948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.174029 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.174498 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.174500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.174605 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.175278 (+ 673us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.175448 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":52,"thread_start_us":126,"threads_started":1}
I20260810 14:35:54.176501 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.149653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49152)
0810 14:35:54.149962 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.149976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.150116 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.150505 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.150507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.167423 (+ 16916us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.167605 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.167617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.169560 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.169563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.170378 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.170396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.171224 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.171231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.171366 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.171879 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.171893 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.172311 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.174421 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.174424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.174426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.174762 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.175114 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.175117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.175119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.175390 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.175976 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.175981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.176090 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.176191 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.176340 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260810 14:35:54.176625 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.149356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.21.43.126:34473)
0810 14:35:54.149502 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.149507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.150050 (+ 543us) server_negotiation.cc:407] Connection header received
0810 14:35:54.150244 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.150247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.150301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.150359 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.167791 (+ 17432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.168571 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.170545 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.170906 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.171013 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.172493 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.172497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.172500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.172531 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.174279 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.174892 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.174896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.174898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.174991 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.175504 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.175508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.175510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.175724 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.175796 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.176311 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.176435 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:54.176729 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:42309
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:34473
--webserver_port=37889
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.126@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
I20260810 14:35:54.191622 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.180892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.0.0.1:59790)
0810 14:35:54.181091 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.181109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.181213 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.181547 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.181553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.181781 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.181918 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.181927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.182978 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.182985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.184084 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.184098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.184834 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.184837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.184936 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.185505 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.185523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.187459 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.189740 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.189747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.189752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.190062 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.190393 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.190397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.190399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.190497 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.190985 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.190989 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.191104 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.191267 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.191433 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 14:35:54.192185 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.180962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.21.43.124:46269)
0810 14:35:54.181083 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.181088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.181163 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:35:54.181344 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.181347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.181400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.181482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.182069 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.182850 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.184248 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.184592 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.184689 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.187632 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.187636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.187639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.187668 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.189598 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.190195 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.190198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.190201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.190291 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.190600 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.190605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.190608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.190839 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.190934 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.191880 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.192023 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:35:54.193467 28388 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.43.124:46269: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680"
member_type: NON_VOTER
last_known_addr {
host: "127.21.43.124"
port: 46269
}
attrs {
promote: true
}
}
dest_uuid: "a11b119fca8349e8920a016bf590fd25"
cas_config_opid_index: 5
I20260810 14:35:54.194145 28388 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } }
I20260810 14:35:54.195657 28997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.197306 28997 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.43.124:46269
I20260810 14:35:54.197480 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.197582 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.198259 29002 master.cc:537] Master@127.21.43.124:46269 shutting down...
I20260810 14:35:54.198649 29137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:35:54.199456 28999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:35:54.202119 28371 consensus_peers.cc:487] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 -> Peer 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124:46269): Couldn't send request to peer 2f1c7c9c1f24452a8515d5951b091680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:35:54.203785 29002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:54.204103 29002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:54.204173 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.195690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.21.43.125:41739)
0810 14:35:54.197769 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.197774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.198714 (+ 940us) server_negotiation.cc:407] Connection header received
0810 14:35:54.198907 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.198911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.198976 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.199044 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:54.199511 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.200349 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.201979 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.202733 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.202816 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.203002 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.203917 (+ 915us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.204016 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986}
I20260810 14:35:54.204198 29002 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: stopping tablet replica
I20260810 14:35:54.204607 28993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.195411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42248)
0810 14:35:54.198635 (+ 3224us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.198653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.198791 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.199257 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.199261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.199279 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:54.199373 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.199378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.200501 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.200504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.201823 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.201835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.203627 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.203630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.203735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.203790 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.204393 (+ 603us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.204490 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3141}
I20260810 14:35:54.205235 29052 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:35:54.205710 28999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 14:35:54.207787 29052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.210279 29082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11b119fca8349e8920a016bf590fd25. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.210422 29082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:35:54.210526 29082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: NON_VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: true } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.210598 29082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:35:54.211284 28999 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: attempting to promote NON_VOTER 50c91f3147b84f338f17ef174822d6a2 to VOTER
I20260810 14:35:54.211782 28999 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } }
I20260810 14:35:54.214097 29052 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 LEARNER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 14:35:54.214434 28371 consensus_peers.cc:487] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 -> Peer 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124:46269): Couldn't send request to peer 2f1c7c9c1f24452a8515d5951b091680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:35:54.214522 28996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 14:35:54.216454 28997 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.216912 29052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.218108 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.218216 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.218338 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.218400 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.218881 29197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.219049 29197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:54.227458 29002 master.cc:552] Master@127.21.43.124:46269 shutdown complete.
I20260810 14:35:54.227535 29002 master_runner.cc:305] Clearing existing system tablet
I20260810 14:35:54.227798 29002 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.227867 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.229362 29002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 14:35:54.230366 29199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.230656 29002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.230789 29002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:35:54.230823 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "2f1c7c9c1f24452a8515d5951b091680"
format_stamp: "Formatted at 2026-08-10 14:35:54 on dist-test-slave-chsk"
server_key: "93f9f9bb1a802ba2a62f294a9b88258b"
server_key_iv: "533bfa793042c3bf848ff84199e78a30"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.232983 29002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:54.235920 29002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:35:54.236050 29002 log.cc:1196] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 14:35:54.236366 29002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Deleting consensus metadata
I20260810 14:35:54.236552 29002 master_runner.cc:315] Copying system tablet from 127.21.43.126:34473
I20260810 14:35:54.237799 29002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: tablet copy: Beginning tablet copy session from remote peer at address 127.21.43.126:34473
I20260810 14:35:54.248642 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.238592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.21.43.126:34473)
0810 14:35:54.238731 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.238736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.239517 (+ 781us) server_negotiation.cc:407] Connection header received
0810 14:35:54.239719 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.239722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.239783 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.239885 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.240688 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.241495 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.242692 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.243015 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.243121 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.244766 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.244770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.244773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.244801 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.246610 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.247212 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.247216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.247218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.247314 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.247649 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.247653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.247655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.247897 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.248001 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.248349 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.248477 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:54.248653 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.238525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49162)
0810 14:35:54.239395 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.239412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.239608 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.239919 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.239922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.240412 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.240543 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.240552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.241621 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.241625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.242557 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.242567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.243379 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.243387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.243492 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.244122 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.244137 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.244611 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.246721 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.246724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.246727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.247012 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.247442 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.247446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.247448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.247543 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.248008 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.248011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.248154 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.248278 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.248438 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":95,"threads_started":1}
I20260810 14:35:54.249481 28410 tablet_copy_service.cc:140] P a11b119fca8349e8920a016bf590fd25: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2f1c7c9c1f24452a8515d5951b091680 ({username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:49162)
I20260810 14:35:54.249578 28410 tablet_copy_service.cc:161] P a11b119fca8349e8920a016bf590fd25: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2f1c7c9c1f24452a8515d5951b091680 at {username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:49162: session id = 2f1c7c9c1f24452a8515d5951b091680-00000000000000000000000000000000
I20260810 14:35:54.250530 28410 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:35:54.251955 29002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:35:54.254513 29002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: tablet copy: Starting download of 0 data blocks...
I20260810 14:35:54.254719 29002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: tablet copy: Starting download of 1 WAL segments...
I20260810 14:35:54.256450 29002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:35:54.258517 28410 tablet_copy_service.cc:342] P a11b119fca8349e8920a016bf590fd25: Request end of tablet copy session 2f1c7c9c1f24452a8515d5951b091680-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:49162
I20260810 14:35:54.258620 28410 tablet_copy_service.cc:433] P a11b119fca8349e8920a016bf590fd25: ending tablet copy session 2f1c7c9c1f24452a8515d5951b091680-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2f1c7c9c1f24452a8515d5951b091680
I20260810 14:35:54.267117 29002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:54.272928 29206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.273275 29209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.273530 29002 server_base.cc:1034] running on GCE node
W20260810 14:35:54.275619 29207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.275942 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:54.276305 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:54.277459 29002 hybrid_clock.cc:648] HybridClock initialized: now 1786372554277449 us; error 35 us; skew 500 ppm
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:54.279376 29002 init.cc:377] Logged in from keytab as oryx/127.21.43.124@KRBTEST.COM (short username oryx)
I20260810 14:35:54.279919 29002 webserver.cc:460] Webserver started at http://127.21.43.124:38189/ using document root <none> and password file <none>
I20260810 14:35:54.280125 29002 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.280174 29002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.281652 29002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:54.282882 29216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.283236 29002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:54.283322 29002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "2f1c7c9c1f24452a8515d5951b091680"
format_stamp: "Formatted at 2026-08-10 14:35:54 on dist-test-slave-chsk"
server_key: "93f9f9bb1a802ba2a62f294a9b88258b"
server_key_iv: "533bfa793042c3bf848ff84199e78a30"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.283412 29002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.290719 28737 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
I20260810 14:35:54.303481 29002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:54.306162 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:54.312011 29002 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.124:46269
I20260810 14:35:54.312042 29269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.124:46269 every 8 connection(s)
I20260810 14:35:54.312683 29002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
I20260810 14:35:54.314694 29270 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:35:54.315986 29270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Bootstrap starting.
I20260810 14:35:54.328504 29270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:35:54.328923 29270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Bootstrap complete.
I20260810 14:35:54.329593 29270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } }
I20260810 14:35:54.329756 29270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f1c7c9c1f24452a8515d5951b091680, State: Initialized, Role: LEARNER
I20260810 14:35:54.329870 29270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } }
I20260810 14:35:54.330248 29277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.330399 29270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:54.330485 29277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: LEARNER
W20260810 14:35:54.333098 29195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:54.333491 29195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:54.333562 29195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:35:54.334331 29288 catalog_manager.cc:1259] Loaded cluster ID: 8fa5849c604f456b97318e724a298f72
I20260810 14:35:54.334394 29288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: loading cluster ID for follower catalog manager: success
I20260810 14:35:54.336644 29288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: acquiring CA information for follower catalog manager: success
I20260810 14:35:54.338256 29288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:35:54.339515 29195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:54.339599 29195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:54.339712 29195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:54.345374 29195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:42309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=37889
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:54.346549 29195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:54.347684 29195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:54.358143 29291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.358181 29290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.358144 29293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.358402 29195 server_base.cc:1034] running on GCE node
I20260810 14:35:54.359009 29195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:54.359805 29195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:54.360991 29195 hybrid_clock.cc:648] HybridClock initialized: now 1786372554360968 us; error 35 us; skew 500 ppm
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:54.364223 29195 init.cc:377] Logged in from keytab as oryx/127.21.43.65@KRBTEST.COM (short username oryx)
I20260810 14:35:54.365386 29195 webserver.cc:460] Webserver started at http://127.21.43.65:37889/ using document root <none> and password file <none>
I20260810 14:35:54.365934 29195 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.366010 29195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.370108 29195 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 14:35:54.372392 29300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.373764 29195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:54.373909 29195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "8efb7e93937744829323a54d7d7d1f20"
format_stamp: "Formatted at 2026-08-10 14:35:52 on dist-test-slave-chsk"
server_key: "4d7cd26d6b6a9691aee4b8fe4c135157"
server_key_iv: "cd26d220bb9bf3e336c2752ede1b4738"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.374472 29195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.395901 29195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:54.399272 29195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:54.399508 29195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:54.400179 29195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:54.401072 29195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:54.401121 29195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.401202 29195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:54.401234 29195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.416493 29195 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:42309
I20260810 14:35:54.416523 29422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:42309 every 8 connection(s)
I20260810 14:35:54.417620 29195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:54.427764 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29195
I20260810 14:35:54.427984 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28598
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
I20260810 14:35:54.436138 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.420296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:47225 (local address 127.21.43.124:46269)
0810 14:35:54.420812 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.420817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.421621 (+ 804us) server_negotiation.cc:407] Connection header received
0810 14:35:54.422754 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.422757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.422800 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.422905 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.424124 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.424838 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426498 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.426740 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426860 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.431246 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.431253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.431258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.431302 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.433168 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434005 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.434009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.434011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.434310 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434625 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.434631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.434635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.434848 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.434925 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.435811 (+ 886us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.435958 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":172,"threads_started":1}
I20260810 14:35:54.436136 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.420136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:39979 (local address 127.21.43.126:34473)
0810 14:35:54.420663 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.420668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.421225 (+ 557us) server_negotiation.cc:407] Connection header received
0810 14:35:54.422377 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.422380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.422454 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.422542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.424144 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.424964 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426461 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.426796 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426892 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.430797 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.430802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.430805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.430837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.433168 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434005 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.434009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.434010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.434124 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434580 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.434586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.434588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.434848 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.434925 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.435817 (+ 892us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.435958 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419}
I20260810 14:35:54.437902 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.420297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.21.43.65:47225)
0810 14:35:54.421526 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.421542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.422158 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.422925 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.422928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.423397 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.423973 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.423988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.424990 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.424994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.426331 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.426342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.426907 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.427176 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.428177 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.428194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.431016 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.433325 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.433332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.433352 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.433876 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.434427 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.434431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.434521 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.435018 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.435025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.435481 (+ 456us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.436328 (+ 847us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.436735 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":395,"spinlock_wait_cycles":18048,"thread_start_us":155,"threads_started":1}
I20260810 14:35:54.437934 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.420297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:39979)
0810 14:35:54.421061 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.421115 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.422132 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.422757 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.422767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.423403 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.423974 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.423990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.425085 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.425088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.426331 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.426342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.427001 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.427005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.427172 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.428078 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.428113 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.430589 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.433325 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.433332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.433352 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.433876 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.434248 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.434254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.434257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.434461 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.435018 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.435025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.435481 (+ 456us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.436831 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.436959 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":404,"spinlock_wait_cycles":59648,"thread_start_us":151,"threads_started":1}
I20260810 14:35:54.439522 29423 heartbeater.cc:347] Connected to a master server at 127.21.43.124:46269
I20260810 14:35:54.440537 29423 heartbeater.cc:464] Registering TS with master...
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:54.441408 29423 heartbeater.cc:511] Master 127.21.43.124:46269 requested a full tablet report, sending...
I20260810 14:35:54.443761 29233 ts_manager.cc:194] Registered new tserver with Master: 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309)
I20260810 14:35:54.445755 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.421167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:33897 (local address 127.21.43.125:41739)
0810 14:35:54.421293 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.421298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.421530 (+ 232us) server_negotiation.cc:407] Connection header received
0810 14:35:54.422483 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.422488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.422553 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.422634 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.424144 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.424958 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426324 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.426665 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426765 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.441748 (+ 14983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.441752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.441754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.441784 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.443859 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.444469 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.444472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.444474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.444565 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.444867 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.444870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.444872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.445079 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.445192 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.445460 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.445589 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:54.445781 29429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.420297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.21.43.65:33897)
0810 14:35:54.421417 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.421431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.422132 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.422757 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.422767 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.423399 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.423974 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.423990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.425096 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.425099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.426187 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.426197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.426911 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.426916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.427127 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.428035 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.439248 (+ 11213us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.441476 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.444076 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.444082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.444087 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.444370 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.444664 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.444669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.444671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.444767 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.445179 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.445182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.445289 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.445403 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.445575 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":376,"spinlock_wait_cycles":23552,"thread_start_us":169,"threads_started":1}
I20260810 14:35:54.446491 29425 heartbeater.cc:347] Connected to a master server at 127.21.43.125:41739
I20260810 14:35:54.446597 29425 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.446807 29425 heartbeater.cc:511] Master 127.21.43.125:41739 requested a full tablet report, sending...
I20260810 14:35:54.447721 29424 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:54.447806 29424 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.447932 29424 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:54.448190 29042 ts_manager.cc:194] Registered new tserver with Master: 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309)
I20260810 14:35:54.448868 28388 ts_manager.cc:194] Re-registered known tserver with Master: 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309)
I20260810 14:35:54.449492 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:39979
I20260810 14:35:54.452576 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:34111
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:34473
--webserver_port=45597
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:54.509909 28886 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
W20260810 14:35:54.571404 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:54.571717 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:54.571798 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:54.575832 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:54.575937 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:54.576040 29431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:54.581938 29431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:34111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=45597
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:54.583274 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:54.584466 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:54.594710 29439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.594745 29436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.594660 29437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.595024 29431 server_base.cc:1034] running on GCE node
I20260810 14:35:54.595557 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:54.596303 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:54.597501 29431 hybrid_clock.cc:648] HybridClock initialized: now 1786372554597483 us; error 39 us; skew 500 ppm
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:54.600390 29431 init.cc:377] Logged in from keytab as oryx/127.21.43.66@KRBTEST.COM (short username oryx)
I20260810 14:35:54.601384 29431 webserver.cc:460] Webserver started at http://127.21.43.66:45597/ using document root <none> and password file <none>
I20260810 14:35:54.602000 29431 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.602121 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.606237 29431 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:54.608491 29447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.609730 29431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:35:54.609876 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "936640a192a6d88fe926f7542a3488cf"
server_key_iv: "537c83057bef4d7518673a94ba2a73d9"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.610460 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.634967 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:54.639724 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:54.639971 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:54.640697 29431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:54.641692 29431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:54.641769 29431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.641849 29431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:54.641894 29431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.653719 29431 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34111
I20260810 14:35:54.653743 29569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34111 every 8 connection(s)
I20260810 14:35:54.654862 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:54.660336 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29431
I20260810 14:35:54.660552 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28742
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
I20260810 14:35:54.674178 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:37981 (local address 127.21.43.126:34473)
0810 14:35:54.657471 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.657476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.658602 (+ 1126us) server_negotiation.cc:407] Connection header received
0810 14:35:54.659913 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.659917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.660003 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.660085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.661779 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.662333 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.664510 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.664865 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.664989 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.669018 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.669027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.669033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.669080 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.671078 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.671864 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.671871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.671876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.671973 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.672461 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.672467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.672472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.672755 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.672850 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.673747 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.673910 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
I20260810 14:35:54.675993 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:37981)
0810 14:35:54.658434 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.658492 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.659669 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.660319 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.660330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.660897 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.661589 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.661608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.662498 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.662505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.664342 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.664360 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.665087 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.665097 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.665413 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.666240 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.666276 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.668769 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.671267 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.671278 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.671302 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.671734 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.672089 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.672095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.672098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.672317 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.672929 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.672939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.673409 (+ 470us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.674400 (+ 991us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.674832 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260810 14:35:54.677435 29571 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:54.677706 29571 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.678328 29571 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:54.678439 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:38713
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:34473
--webserver_port=40293
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:54.678480 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:35463 (local address 127.21.43.125:41739)
0810 14:35:54.657923 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.657928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.659357 (+ 1429us) server_negotiation.cc:407] Connection header received
0810 14:35:54.667722 (+ 8365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.667729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.667811 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.667916 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.668666 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.669499 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.670764 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.671114 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.671220 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.673872 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.673878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.673880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.673911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.675818 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.676761 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.676767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.676773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.676881 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.677268 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.677274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.677276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.677562 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.677649 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.678187 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.678324 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:54.679580 28388 ts_manager.cc:194] Re-registered known tserver with Master: 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:54.680372 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.21.43.66:35463)
0810 14:35:54.659232 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.659245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.659685 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.667989 (+ 8304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.667997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.668280 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.668478 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.668489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.669709 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.669715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.670573 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.670582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.671397 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.671401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.671505 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.672050 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.672065 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.673683 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.676036 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.676039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.676041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.676627 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.677028 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.677033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.677036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.677142 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.677834 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.677839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.677955 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.679309 (+ 1354us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.679483 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"mutex_wait_us":179,"thread_start_us":125,"threads_started":1}
I20260810 14:35:54.680437 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:37981
I20260810 14:35:54.680568 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:54947 (local address 127.21.43.124:46269)
0810 14:35:54.657322 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.657327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.659098 (+ 1771us) server_negotiation.cc:407] Connection header received
0810 14:35:54.668080 (+ 8982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.668087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.668175 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.668258 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.670396 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.671219 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.672464 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.672864 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.672959 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.676220 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.676224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.676226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.676257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.678144 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.678931 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.678937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.678940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.679052 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.679402 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.679405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.679407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.679668 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.679748 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.680226 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.680370 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:35:54.681110 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.657828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.21.43.66:54947)
0810 14:35:54.658971 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.658984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.659669 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.669916 (+ 10247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.669920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.670109 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.670235 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.670242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.671375 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.671379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.672310 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.672322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.673145 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.673148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.673255 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.673771 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.673789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.676047 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.678525 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.678531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.678535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.678793 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.679193 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.679196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.679198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.679288 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.679858 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.679861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.679977 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.680768 (+ 791us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.680935 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":37,"spinlock_wait_cycles":29952,"thread_start_us":223,"threads_started":1}
I20260810 14:35:54.681635 29572 heartbeater.cc:347] Connected to a master server at 127.21.43.125:41739
I20260810 14:35:54.681720 29570 heartbeater.cc:347] Connected to a master server at 127.21.43.124:46269
I20260810 14:35:54.681790 29570 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.681730 29572 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.682080 29570 heartbeater.cc:511] Master 127.21.43.124:46269 requested a full tablet report, sending...
I20260810 14:35:54.682121 29572 heartbeater.cc:511] Master 127.21.43.125:41739 requested a full tablet report, sending...
I20260810 14:35:54.682837 29233 ts_manager.cc:194] Registered new tserver with Master: 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:54.682837 29042 ts_manager.cc:194] Registered new tserver with Master: 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:54.796048 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.792478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.21.43.124:46269)
0810 14:35:54.792635 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.792640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.792750 (+ 110us) server_negotiation.cc:407] Connection header received
0810 14:35:54.792926 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.792931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.792992 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.793099 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:54.793386 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.793899 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.794953 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.795467 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.795589 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.795749 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.795828 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.795902 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:35:54.796051 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.792374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.0.0.1:59798)
0810 14:35:54.792635 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.792653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.792811 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.793129 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.793132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.793153 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:54.793271 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.793277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.794017 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.794021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.794818 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.794828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.795601 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.795605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.795694 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.795728 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.795814 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.795923 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260810 14:35:54.796993 29243 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 14:35:54.797463 29277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11b119fca8349e8920a016bf590fd25. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.797653 29277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: LEARNER
W20260810 14:35:54.803915 29577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:54.804193 29577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:54.804282 29577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:54.808629 29577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:54.808691 29577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:54.808794 29577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:54.814522 29577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:38713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=40293
--tserver_master_addrs=127.21.43.126:34473,127.21.43.125:41739,127.21.43.124:46269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:54.816017 29577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:54.817222 29577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:54.828409 29587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:54.828343 29584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.828495 29577 server_base.cc:1034] running on GCE node
W20260810 14:35:54.828343 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:54.829162 29577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:54.829913 29577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:54.831084 29577 hybrid_clock.cc:648] HybridClock initialized: now 1786372554831062 us; error 37 us; skew 500 ppm
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:54.834381 29577 init.cc:377] Logged in from keytab as oryx/127.21.43.67@KRBTEST.COM (short username oryx)
I20260810 14:35:54.835462 29577 webserver.cc:460] Webserver started at http://127.21.43.67:40293/ using document root <none> and password file <none>
I20260810 14:35:54.836082 29577 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:54.836176 29577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:54.841041 29577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 14:35:54.843561 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.845088 29577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:35:54.845271 29577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "26fabb5275c345c78f1e0d469de75525"
format_stamp: "Formatted at 2026-08-10 14:35:53 on dist-test-slave-chsk"
server_key: "131b0775140bc41f25ce99765f1feef3"
server_key_iv: "50dde72f716a4aef98254313dbaf7086"
server_key_version: "encryptionkey@0"
I20260810 14:35:54.845942 29577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:54.861248 29577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:54.864770 29577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:54.865020 29577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:54.865717 29577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:54.866675 29577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:54.866748 29577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.866823 29577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:54.866876 29577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:54.877045 29577 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:38713
I20260810 14:35:54.877074 29724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:38713 every 8 connection(s)
I20260810 14:35:54.878237 29577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:54.887845 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29577
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.125@KRBTEST.COM
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
Aug 10 14:35:54 dist-test-slave-chsk krb5kdc[28341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372554, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.124@KRBTEST.COM
I20260810 14:35:54.895334 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.880338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:41903 (local address 127.21.43.125:41739)
0810 14:35:54.880482 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.880486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.881552 (+ 1066us) server_negotiation.cc:407] Connection header received
0810 14:35:54.882897 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.882901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.882956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.883011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.884723 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885286 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.886961 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887388 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887483 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.890842 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.890849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.890856 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.890892 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.892739 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.893462 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.893466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.893468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.893556 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.893969 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.893972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.893974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.894186 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.894255 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.894989 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.895124 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:35:54.896713 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.880416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:33125 (local address 127.21.43.124:46269)
0810 14:35:54.880545 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.880549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.881751 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 14:35:54.882877 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.882881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.882947 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.883024 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.884716 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885545 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.886953 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887318 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887422 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.893015 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.893020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.893024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.893058 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.894741 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.895331 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.895338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.895339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.895427 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.895742 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.895748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.895753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.895976 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.896059 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.896383 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.896516 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:35:54.896984 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.882185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:59347 (local address 127.21.43.126:34473)
0810 14:35:54.882328 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.882338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.882440 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:35:54.882897 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.882901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.882956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.883011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.884724 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885545 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.886913 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887271 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.892027 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.892036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.892040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.892075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.894414 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.894940 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.894946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.894949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.895033 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.895386 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.895392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.895394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.895626 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.895735 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.896670 (+ 935us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.896813 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:35:54.897575 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.880816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.21.43.67:41903)
0810 14:35:54.881381 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.881435 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.882663 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.883255 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.883264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.883911 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.884544 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.884552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885443 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.885445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.886804 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.886814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887634 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.887822 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.888521 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.888544 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.890613 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.892884 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.892891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.892912 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.893344 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.893673 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.893676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.893678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.893861 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.894346 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.894353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.894698 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.895517 (+ 819us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.895891 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":215,"spinlock_wait_cycles":55296,"thread_start_us":602,"threads_started":1}
I20260810 14:35:54.897795 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.882022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:59347)
0810 14:35:54.882347 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.882360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.882663 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.883255 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.883264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.883906 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.884532 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.884547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885714 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.885717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.886780 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.886790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887446 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.887684 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.888840 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.888859 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.891530 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.894558 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.894561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.894563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.894830 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.895173 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.895176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.895177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.895279 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.896008 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.896014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.896117 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.896260 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.897525 (+ 1265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":148,"spinlock_wait_cycles":5760,"thread_start_us":139,"threads_started":1}
I20260810 14:35:54.897660 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.880816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.21.43.67:33125)
0810 14:35:54.881654 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.881667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.882661 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.883258 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.883265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.883911 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:54.884542 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.884552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.885675 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.885679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.886804 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.886814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.887582 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.887585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.887709 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.888665 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:54.888686 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:54.892826 (+ 4140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:54.894866 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.894872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.894875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.895211 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.895550 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:54.895553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:54.895555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:54.895636 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:54.896115 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:54.896117 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:54.896189 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.897307 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.897453 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":242,"spinlock_wait_cycles":44928,"thread_start_us":182,"threads_started":1}
I20260810 14:35:54.899040 29725 heartbeater.cc:347] Connected to a master server at 127.21.43.124:46269
I20260810 14:35:54.899353 29725 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.899981 29725 heartbeater.cc:511] Master 127.21.43.124:46269 requested a full tablet report, sending...
I20260810 14:35:54.900574 29726 heartbeater.cc:347] Connected to a master server at 127.21.43.126:34473
I20260810 14:35:54.900663 29726 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.900853 29726 heartbeater.cc:511] Master 127.21.43.126:34473 requested a full tablet report, sending...
I20260810 14:35:54.901051 29727 heartbeater.cc:347] Connected to a master server at 127.21.43.125:41739
I20260810 14:35:54.901127 29727 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:54.901292 29727 heartbeater.cc:511] Master 127.21.43.125:41739 requested a full tablet report, sending...
I20260810 14:35:54.901300 29233 ts_manager.cc:194] Registered new tserver with Master: 26fabb5275c345c78f1e0d469de75525 (127.21.43.67:38713)
I20260810 14:35:54.902019 29042 ts_manager.cc:194] Registered new tserver with Master: 26fabb5275c345c78f1e0d469de75525 (127.21.43.67:38713)
I20260810 14:35:54.902915 28388 ts_manager.cc:194] Re-registered known tserver with Master: 26fabb5275c345c78f1e0d469de75525 (127.21.43.67:38713)
I20260810 14:35:54.903383 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.891005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.21.43.125:41739)
0810 14:35:54.891575 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.891581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.892173 (+ 592us) server_negotiation.cc:407] Connection header received
0810 14:35:54.892220 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.892223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.892278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.892360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.893462 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.894253 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.895504 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.895845 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.895947 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.897810 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.897813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.897816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.897844 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.901014 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.901588 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.901592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.901594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.901687 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.902023 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.902027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.902029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.902436 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.902583 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.902943 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.903081 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":98,"threads_started":1}
I20260810 14:35:54.903798 28388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:59347
I20260810 14:35:54.908471 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.891305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.21.43.124:46269)
0810 14:35:54.896875 (+ 5570us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.896880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.899279 (+ 2399us) server_negotiation.cc:407] Connection header received
0810 14:35:54.899484 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.899487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.899538 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.899599 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.900613 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.901393 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.902710 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.903060 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.903169 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.905055 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.905059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.905061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.905092 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.906530 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.906987 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.906991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.906992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.907085 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.907456 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.907459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.907460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.907660 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.907837 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.908115 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.908228 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5513,"thread_start_us":99,"threads_started":1}
I20260810 14:35:54.911507 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.890783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.21.43.126:34473)
0810 14:35:54.893887 (+ 3104us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.893892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.893909 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:35:54.893976 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.893979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.898604 (+ 4625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.898696 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:54.900021 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.902450 (+ 2429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.905170 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.905491 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.905603 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.907413 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:54.907419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:54.907425 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:54.907458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:54.909567 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.910016 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.910020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.910021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.910121 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:54.910444 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:54.910447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:54.910449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:54.910657 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:54.910810 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.911053 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.911227 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2670,"thread_start_us":73,"threads_started":1}
I20260810 14:35:54.914705 28388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:54.917999 28388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:54.920861 29578 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: NON_VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: true } } }
I20260810 14:35:54.921042 29578 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.924813 29747 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: attempting to promote NON_VOTER 2f1c7c9c1f24452a8515d5951b091680 to VOTER
I20260810 14:35:54.925177 29747 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } }
I20260810 14:35:54.926808 29052 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:35:54.927219 29745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 14:35:54.928128 29242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 LEARNER]: Refusing update from remote peer a11b119fca8349e8920a016bf590fd25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:35:54.928911 29746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 14:35:54.929287 29578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.929939 29052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.930959 29578 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.931056 29578 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.931244 29578 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.931316 29578 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.931890 29756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.932021 29756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:54.933818 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.933537 29242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.933923 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:54.935503 29277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11b119fca8349e8920a016bf590fd25" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:54.935632 29277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:54.944039 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.938084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.21.43.66:34111)
0810 14:35:54.938419 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.938431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.939330 (+ 899us) server_negotiation.cc:407] Connection header received
0810 14:35:54.939459 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.939463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.939663 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.939819 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:54.940297 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.941379 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.942815 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.943472 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.943595 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.943764 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.943818 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.943904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260810 14:35:54.944139 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.937059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42309 (local address 127.0.0.1:45346)
0810 14:35:54.937383 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.937398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.937476 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.939860 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.939863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.939894 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:54.940001 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.940007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.941139 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.941142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.942585 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.942597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.943717 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.943721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.943804 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.943829 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.943878 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.943955 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":186,"spinlock_wait_cycles":4864}
I20260810 14:35:54.944013 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.937808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34111 (local address 127.0.0.1:39092)
0810 14:35:54.939223 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.939235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.939341 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.939861 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.939864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.939896 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:54.940176 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.940183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.941501 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.941504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.942685 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.942694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.943596 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.943600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.943682 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.943705 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.943771 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.943848 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":50,"spinlock_wait_cycles":9088,"thread_start_us":120,"threads_started":1}
I20260810 14:35:54.944583 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.937392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.21.43.65:42309)
0810 14:35:54.939419 (+ 2027us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.939433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.939458 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:35:54.939534 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.939538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.939701 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.939959 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:54.940102 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.941000 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.942742 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.943585 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.944103 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.944285 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.944342 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.944428 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1927,"thread_start_us":98,"threads_started":1}
I20260810 14:35:54.946684 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.937637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38713 (local address 127.0.0.1:44364)
0810 14:35:54.938819 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.938836 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:54.938942 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:54.939559 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:54.939562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:54.939580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:54.939701 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.939710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.941115 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.941118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.943555 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:54.943567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.946252 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.946256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:54.946350 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.946375 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:35:54.946469 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 14:35:54.946550 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":56,"thread_start_us":128,"threads_started":1}
I20260810 14:35:54.946743 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.938094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.21.43.67:38713)
0810 14:35:54.938625 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:54.938634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:54.938944 (+ 310us) server_negotiation.cc:407] Connection header received
0810 14:35:54.939110 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:54.939115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:54.939396 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:54.939589 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:54.939848 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.940967 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.943725 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:54.946104 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:54.946228 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:54.946364 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:54.946497 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 14:35:54.946588 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":122,"threads_started":1}
I20260810 14:35:54.947592 29500 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 (DEFAULT_TABLE table=test-table [id=4fdf3931865c43359a2e29aae72cc250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:54.947575 29352 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 (DEFAULT_TABLE table=test-table [id=4fdf3931865c43359a2e29aae72cc250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:54.948630 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ba5e91a8b4d49b474cd54225a7fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:54.948606 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ba5e91a8b4d49b474cd54225a7fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:54.949729 29645 tablet_service.cc:1467] Processing CreateTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 (DEFAULT_TABLE table=test-table [id=4fdf3931865c43359a2e29aae72cc250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:54.950760 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9ba5e91a8b4d49b474cd54225a7fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:54.956163 29765 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Bootstrap starting.
I20260810 14:35:54.959056 29767 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Bootstrap starting.
I20260810 14:35:54.959354 29765 tablet_bootstrap.cc:654] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:54.960177 29766 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Bootstrap starting.
I20260810 14:35:54.960438 29765 log.cc:824] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:54.961470 29767 tablet_bootstrap.cc:654] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:54.962419 29767 log.cc:824] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:54.962507 29766 tablet_bootstrap.cc:654] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:54.962603 29765 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: No bootstrap required, opened a new log
I20260810 14:35:54.962826 29765 ts_tablet_manager.cc:1397] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 14:35:54.963490 29766 log.cc:824] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:54.965590 29767 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: No bootstrap required, opened a new log
I20260810 14:35:54.965819 29767 ts_tablet_manager.cc:1397] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260810 14:35:54.966238 29766 tablet_bootstrap.cc:492] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: No bootstrap required, opened a new log
I20260810 14:35:54.966439 29766 ts_tablet_manager.cc:1397] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 14:35:54.967901 29765 raft_consensus.cc:348] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.968194 29765 raft_consensus.cc:374] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:54.968258 29765 raft_consensus.cc:729] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfc628b3d3c4c1787ff2d48c3b2a26f, State: Initialized, Role: FOLLOWER
I20260810 14:35:54.968784 29765 consensus_queue.cc:260] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.969664 29571 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
I20260810 14:35:54.970544 29766 raft_consensus.cc:348] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.970851 29766 raft_consensus.cc:374] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:54.970913 29766 raft_consensus.cc:729] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efb7e93937744829323a54d7d7d1f20, State: Initialized, Role: FOLLOWER
I20260810 14:35:54.970762 29767 raft_consensus.cc:348] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.970992 29767 raft_consensus.cc:374] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:54.971035 29767 raft_consensus.cc:729] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fabb5275c345c78f1e0d469de75525, State: Initialized, Role: FOLLOWER
I20260810 14:35:54.971427 29767 consensus_queue.cc:260] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.971427 29766 consensus_queue.cc:260] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:54.972448 29766 ts_tablet_manager.cc:1428] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:35:54.972448 29767 ts_tablet_manager.cc:1428] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 14:35:54.972798 29424 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
I20260810 14:35:54.974455 29765 ts_tablet_manager.cc:1428] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 14:35:54.974710 29726 heartbeater.cc:503] Master 127.21.43.126:34473 was elected leader, sending a full tablet report...
I20260810 14:35:55.037505 29773 raft_consensus.cc:482] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:55.037727 29773 raft_consensus.cc:504] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:55.039120 29773 leader_election.cc:288] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309), 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:55.043026 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.039547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:60667 (local address 127.21.43.66:34111)
0810 14:35:55.039702 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.039707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.039811 (+ 104us) server_negotiation.cc:407] Connection header received
0810 14:35:55.039987 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.039990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.040099 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.040191 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:55.040462 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.040973 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042036 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.042450 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.042640 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.042830 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.042899 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 14:35:55.043043 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.039485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34111 (local address 127.21.43.67:60667)
0810 14:35:55.039711 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:55.039728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:55.039865 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:55.040205 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:55.040208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:55.040224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:55.040345 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:55.040358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.041088 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.041091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:55.041897 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:55.041905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.042583 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:55.042685 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.042717 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:35:55.042814 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:35:55.042886 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 14:35:55.043226 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.039554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42309 (local address 127.21.43.67:51203)
0810 14:35:55.039757 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:55.039780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:55.039891 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:55.040253 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:55.040256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:55.040274 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:55.040400 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:55.040407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.041193 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.041197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:55.042019 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:55.042028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.042740 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:55.042853 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.042875 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:55.042978 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:35:55.043058 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260810 14:35:55.043249 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.039624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:51203 (local address 127.21.43.65:42309)
0810 14:35:55.039759 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.039765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.039834 (+ 69us) server_negotiation.cc:407] Connection header received
0810 14:35:55.040074 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.040079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.040143 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.040238 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:55.040520 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.041067 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042167 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.042605 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.042746 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.042799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.043012 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.043089 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:35:55.043848 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ba5e91a8b4d49b474cd54225a7fe3" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" is_pre_election: true
I20260810 14:35:55.044031 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ba5e91a8b4d49b474cd54225a7fe3" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efb7e93937744829323a54d7d7d1f20" is_pre_election: true
I20260810 14:35:55.044203 29523 raft_consensus.cc:2454] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 0.
I20260810 14:35:55.044276 29373 raft_consensus.cc:2454] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 0.
I20260810 14:35:55.044708 29599 leader_election.cc:302] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26fabb5275c345c78f1e0d469de75525, 3cfc628b3d3c4c1787ff2d48c3b2a26f, 8efb7e93937744829323a54d7d7d1f20; no voters:
I20260810 14:35:55.044993 29773 raft_consensus.cc:2781] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:55.045083 29773 raft_consensus.cc:482] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:55.045121 29773 raft_consensus.cc:3037] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:55.046314 29773 raft_consensus.cc:504] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:55.046716 29773 leader_election.cc:288] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 election: Requested vote from peers 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309), 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:55.047088 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ba5e91a8b4d49b474cd54225a7fe3" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
I20260810 14:35:55.047088 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9ba5e91a8b4d49b474cd54225a7fe3" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efb7e93937744829323a54d7d7d1f20"
I20260810 14:35:55.047255 29523 raft_consensus.cc:3037] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:55.047255 29373 raft_consensus.cc:3037] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:55.048658 29373 raft_consensus.cc:2454] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 1.
I20260810 14:35:55.048651 29523 raft_consensus.cc:2454] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 1.
I20260810 14:35:55.048974 29597 leader_election.cc:302] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fabb5275c345c78f1e0d469de75525, 3cfc628b3d3c4c1787ff2d48c3b2a26f; no voters:
I20260810 14:35:55.049163 29773 raft_consensus.cc:2781] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:55.049448 29773 raft_consensus.cc:686] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 1 LEADER]: Becoming Leader. State: Replica: 26fabb5275c345c78f1e0d469de75525, State: Running, Role: LEADER
I20260810 14:35:55.049880 29773 consensus_queue.cc:237] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } }
I20260810 14:35:55.053597 28388 catalog_manager.cc:5495] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26fabb5275c345c78f1e0d469de75525 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "26fabb5275c345c78f1e0d469de75525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } health_report { overall_health: HEALTHY } } }
I20260810 14:35:55.077483 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.073688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.21.43.67:38713)
0810 14:35:55.073843 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.073848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.073864 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:55.074005 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.074008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.074063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.074181 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:55.074434 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.074989 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.075911 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.076131 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.076242 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.076587 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:55.077051 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:55.077191 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.077255 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.077325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:35:55.084370 29373 raft_consensus.cc:1261] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Refusing update from remote peer 26fabb5275c345c78f1e0d469de75525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:55.084410 29523 raft_consensus.cc:1261] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Refusing update from remote peer 26fabb5275c345c78f1e0d469de75525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:55.085320 29773 consensus_queue.cc:1035] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:55.085561 29774 consensus_queue.cc:1035] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:55.096827 29779 mvcc.cc:205] Tried to move back new op lower bound from 7316981985616728064 to 7316981985487695872. Current Snapshot: MvccSnapshot[applied={T|T < 7316981985616728064}]
I20260810 14:35:55.106535 28388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:55.106731 28388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:55.108934 29781 mvcc.cc:205] Tried to move back new op lower bound from 7316981985616728064 to 7316981985487695872. Current Snapshot: MvccSnapshot[applied={T|T < 7316981985616728064 or (T in {7316981985616728064})}]
I20260810 14:35:55.111881 28388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a11b119fca8349e8920a016bf590fd25: Sending DeleteTablet for 3 replicas of tablet ed9ba5e91a8b4d49b474cd54225a7fe3
I20260810 14:35:55.112962 29352 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:55 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:45346
I20260810 14:35:55.113574 29788 tablet_replica.cc:330] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: stopping tablet replica
I20260810 14:35:55.113978 29788 raft_consensus.cc:2229] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:55.114280 29645 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:55 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:44364
I20260810 14:35:55.114470 29788 raft_consensus.cc:2258] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:55.115331 29791 tablet_replica.cc:330] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: stopping tablet replica
I20260810 14:35:55.115424 29500 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9ba5e91a8b4d49b474cd54225a7fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:55 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:39092
I20260810 14:35:55.115787 29791 raft_consensus.cc:2229] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:55.116215 29791 raft_consensus.cc:2258] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:55.116534 29788 ts_tablet_manager.cc:1905] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:55.118032 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28354
I20260810 14:35:55.120499 29791 ts_tablet_manager.cc:1905] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:55.121237 29788 ts_tablet_manager.cc:1918] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:55.121381 29788 log.cc:1196] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/ed9ba5e91a8b4d49b474cd54225a7fe3
I20260810 14:35:55.121821 29788 ts_tablet_manager.cc:1939] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 8efb7e93937744829323a54d7d7d1f20: Deleting consensus metadata
I20260810 14:35:55.124198 29791 ts_tablet_manager.cc:1918] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:55.124346 29791 log.cc:1196] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/ed9ba5e91a8b4d49b474cd54225a7fe3
I20260810 14:35:55.124719 29791 ts_tablet_manager.cc:1939] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 26fabb5275c345c78f1e0d469de75525: Deleting consensus metadata
I20260810 14:35:55.127441 29792 tablet_replica.cc:330] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: stopping tablet replica
I20260810 14:35:55.127851 29792 raft_consensus.cc:2229] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:55.128233 29792 raft_consensus.cc:2258] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 14:35:55.129655 29596 connection.cc:535] server connection from 127.0.0.1:44364 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:35:55.130239 29304 connection.cc:535] server connection from 127.0.0.1:45346 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:35:55.130287 29792 ts_tablet_manager.cc:1905] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:55.134377 29792 ts_tablet_manager.cc:1918] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:55.134632 29792 log.cc:1196] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/ed9ba5e91a8b4d49b474cd54225a7fe3
I20260810 14:35:55.135233 29792 ts_tablet_manager.cc:1939] T ed9ba5e91a8b4d49b474cd54225a7fe3 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting consensus metadata
W20260810 14:35:55.136790 29448 connection.cc:441] server connection from 127.0.0.1:39092 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39092 (request call id 1) could send its response
I20260810 14:35:55.140443 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.114709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.21.43.125:41739)
0810 14:35:55.114944 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.114949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.128946 (+ 13997us) server_negotiation.cc:407] Connection header received
0810 14:35:55.129244 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.129248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.129324 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.129389 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:55.130710 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.131623 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.133236 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.133590 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.133676 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.136417 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:55.136421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:55.136424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:55.136455 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:55.138330 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.138933 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.138937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.138939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.139029 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.139401 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.139404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.139407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.139659 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:55.139796 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.140134 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.140275 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260810 14:35:55.147845 29170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.136196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.21.43.125:41739)
0810 14:35:55.136330 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.136336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.137155 (+ 819us) server_negotiation.cc:407] Connection header received
0810 14:35:55.137409 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.137415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.137471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.137569 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:55.138620 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.139474 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.140999 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.141331 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.141404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.144069 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:55.144074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:55.144076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:55.144105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:55.145895 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.146376 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.146379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.146381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.146466 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.146802 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.146805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.146807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.147028 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:55.147174 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.147548 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.147673 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:35:55.149972 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.138973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.21.43.124:46269)
0810 14:35:55.139107 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.139112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.139698 (+ 586us) server_negotiation.cc:407] Connection header received
0810 14:35:55.139877 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.139880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.139935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.140000 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:55.140849 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.141627 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.142892 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.143525 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.143661 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.145765 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:55.145768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:55.145771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:55.145799 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:55.147829 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.148334 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.148337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.148339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.148468 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.148807 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.148816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.148820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.149093 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:55.149208 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.149639 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.149801 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":53}
I20260810 14:35:55.151873 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.114818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.21.43.124:46269)
0810 14:35:55.115079 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:55.115084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:55.141929 (+ 26845us) server_negotiation.cc:407] Connection header received
0810 14:35:55.142073 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:55.142079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:55.142137 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:55.142222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:55.143000 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.144274 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.145547 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:55.145897 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:55.145980 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:55.148278 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:55.148282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:55.148288 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:55.148322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:55.150110 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.150616 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.150619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.150621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.150715 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:55.151018 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:55.151021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:55.151022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:55.151313 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:55.151420 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:55.151643 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 14:35:55.151732 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":336,"server-negotiator.queue_time_us":147}
I20260810 14:35:56.100049 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.099527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:42847)
0810 14:35:56.099807 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.099898 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
W20260810 14:35:56.100389 29424 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:34473 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.107547 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.106921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:56305)
0810 14:35:56.107301 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.107411 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260810 14:35:56.107937 29726 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:34473 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.114212 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.113678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:59423)
0810 14:35:56.114014 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.114111 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
W20260810 14:35:56.114535 29571 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:34473 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.612731 29804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a11b119fca8349e8920a016bf590fd25)
I20260810 14:35:56.613063 29804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } }
I20260810 14:35:56.613310 29805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a11b119fca8349e8920a016bf590fd25)
I20260810 14:35:56.613462 29805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } }
I20260810 14:35:56.614482 29805 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473), 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125:41739)
I20260810 14:35:56.615083 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.614201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49668)
0810 14:35:56.614798 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.614897 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":70,"threads_started":1}
I20260810 14:35:56.615334 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.614842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49684)
0810 14:35:56.615133 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.615236 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
W20260810 14:35:56.615401 29220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.615955 29804 leader_election.cc:288] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473), 2f1c7c9c1f24452a8515d5951b091680 (127.21.43.124:46269)
W20260810 14:35:56.615965 29028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.616394 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.616030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49688)
0810 14:35:56.616259 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.616314 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260810 14:35:56.616572 29220 leader_election.cc:334] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473): Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.617008 29811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.616582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49698)
0810 14:35:56.616887 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.616948 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
W20260810 14:35:56.617132 29028 leader_election.cc:334] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473): Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.619315 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.614290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:41739 (local address 127.0.0.1:42274)
0810 14:35:56.614885 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.614904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:56.615042 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:56.615527 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:56.615530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:56.615552 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:56.615675 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.615682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.616757 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.616760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.618008 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.618016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.618885 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.618887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.618958 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.618981 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:35:56.619068 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:35:56.619130 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":60,"threads_started":1}
I20260810 14:35:56.619323 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.614385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.21.43.125:41739)
0810 14:35:56.614742 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:56.614746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:56.614982 (+ 236us) server_negotiation.cc:407] Connection header received
0810 14:35:56.615249 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:56.615252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:56.615299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:56.615358 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:56.615805 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.616613 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.618145 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.618747 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.618863 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.618925 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:56.619086 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 14:35:56.619224 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260810 14:35:56.619781 29052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f1c7c9c1f24452a8515d5951b091680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50c91f3147b84f338f17ef174822d6a2" is_pre_election: true
I20260810 14:35:56.619951 29052 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1c7c9c1f24452a8515d5951b091680 in term 1.
I20260810 14:35:56.620365 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.615327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:46269 (local address 127.0.0.1:59828)
0810 14:35:56.615488 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.615503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:56.615584 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:56.616719 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:56.616722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:56.616745 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:56.616823 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.616829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.617876 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.617878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.619082 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.619093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.619960 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.619964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.620073 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.620095 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:56.620175 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:35:56.620257 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260810 14:35:56.620318 29219 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f1c7c9c1f24452a8515d5951b091680, 50c91f3147b84f338f17ef174822d6a2; no voters: a11b119fca8349e8920a016bf590fd25
I20260810 14:35:56.620580 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.615715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.21.43.124:46269)
0810 14:35:56.616484 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:56.616493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:56.616508 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:35:56.616550 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:56.616553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:56.616615 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:56.616713 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:56.616980 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.617738 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.619234 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.619834 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.619959 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.620016 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:56.620211 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:35:56.620293 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":695,"thread_start_us":90,"threads_started":1}
I20260810 14:35:56.620584 29805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:35:56.620738 29805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a11b119fca8349e8920a016bf590fd25)
I20260810 14:35:56.620790 29805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:35:56.621006 29242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50c91f3147b84f338f17ef174822d6a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f1c7c9c1f24452a8515d5951b091680" is_pre_election: true
I20260810 14:35:56.622037 29805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } }
I20260810 14:35:56.622318 29242 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50c91f3147b84f338f17ef174822d6a2 in current term 2: Already voted for candidate 2f1c7c9c1f24452a8515d5951b091680 in this term.
I20260810 14:35:56.622488 29805 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 election: Requested vote from peers a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473), 50c91f3147b84f338f17ef174822d6a2 (127.21.43.125:41739)
I20260810 14:35:56.622869 29052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f1c7c9c1f24452a8515d5951b091680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50c91f3147b84f338f17ef174822d6a2"
I20260810 14:35:56.623013 29052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:35:56.623338 29025 leader_election.cc:302] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50c91f3147b84f338f17ef174822d6a2; no voters: 2f1c7c9c1f24452a8515d5951b091680, a11b119fca8349e8920a016bf590fd25
I20260810 14:35:56.624209 29052 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1c7c9c1f24452a8515d5951b091680 in term 2.
I20260810 14:35:56.624377 29804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:35:56.624545 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.622653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49710)
0810 14:35:56.622752 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.624467 (+ 1715us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 14:35:56.624536 29219 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1c7c9c1f24452a8515d5951b091680, 50c91f3147b84f338f17ef174822d6a2; no voters:
I20260810 14:35:56.624747 29805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:35:56.624984 29805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [term 2 LEADER]: Becoming Leader. State: Replica: 2f1c7c9c1f24452a8515d5951b091680, State: Running, Role: LEADER
I20260810 14:35:56.625269 29805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } }
I20260810 14:35:56.625464 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.625189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49720)
0810 14:35:56.625338 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.625381 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 14:35:56.625574 29220 leader_election.cc:334] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473): Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
I20260810 14:35:56.626552 29815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1c7c9c1f24452a8515d5951b091680. Latest consensus state: current_term: 2 leader_uuid: "2f1c7c9c1f24452a8515d5951b091680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:56.626652 29815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:56.626955 29817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:56.629570 29817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:56.629983 29817 catalog_manager.cc:1259] Loaded cluster ID: 8fa5849c604f456b97318e724a298f72
I20260810 14:35:56.630055 29817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:56.630620 29817 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:56.631071 29817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Loaded TSK: 0
I20260810 14:35:56.631891 29817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:35:56.680356 29233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:56.683828 29233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:56.687249 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.686900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49738)
0810 14:35:56.687019 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.687111 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 14:35:56.687314 29052 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [term 2 FOLLOWER]: Refusing update from remote peer 2f1c7c9c1f24452a8515d5951b091680: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 14:35:56.687871 29815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 14:35:56.688091 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.687841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.0.0.1:49754)
0810 14:35:56.687944 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.687993 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:35:56.688246 29220 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 -> Peer a11b119fca8349e8920a016bf590fd25 (127.21.43.126:34473): Couldn't send request to peer a11b119fca8349e8920a016bf590fd25. Status: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:35:56.689456 29804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1c7c9c1f24452a8515d5951b091680. Latest consensus state: current_term: 2 leader_uuid: "2f1c7c9c1f24452a8515d5951b091680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:56.689621 29804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:56.690831 29805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2f1c7c9c1f24452a8515d5951b091680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:56.690965 29805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:56.691434 29804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2f1c7c9c1f24452a8515d5951b091680" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a11b119fca8349e8920a016bf590fd25" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34473 } } peers { permanent_uuid: "50c91f3147b84f338f17ef174822d6a2" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 41739 } attrs { promote: false } } peers { permanent_uuid: "2f1c7c9c1f24452a8515d5951b091680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 46269 } attrs { promote: false } } }
I20260810 14:35:56.691568 29804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50c91f3147b84f338f17ef174822d6a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:35:56.701568 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.697534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42309 (local address 127.0.0.1:45362)
0810 14:35:56.697715 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.697732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:56.697814 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:56.698248 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:56.698250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:56.698268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:56.698371 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.698378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.699402 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.699406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.700340 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.700349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.701147 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.701151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.701252 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.701279 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:35:56.701356 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:35:56.701428 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 14:35:56.701625 29824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.697649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.21.43.65:42309)
0810 14:35:56.698026 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:56.698029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:56.698042 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:56.698087 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:56.698089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:56.698126 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:56.698234 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:56.698483 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.699265 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.700471 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.701037 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.701153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.701225 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:56.701389 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:35:56.701475 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":154,"threads_started":1}
I20260810 14:35:56.702311 29352 tablet_service.cc:1467] Processing CreateTablet for tablet cd89611e0ad444a4a7014b4d31620d96 (DEFAULT_TABLE table=test-table [id=7aed97cfded34460ad7d8bee765cce91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:56.702647 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd89611e0ad444a4a7014b4d31620d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:56.703394 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.698737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34111 (local address 127.0.0.1:39100)
0810 14:35:56.699179 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.699195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:56.699273 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:56.699677 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:56.699680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:56.699699 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:56.699796 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.699803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.700794 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.700799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.702003 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.702014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.702890 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.702894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.702988 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.703032 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 14:35:56.703159 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:35:56.703245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":6,"thread_start_us":101,"threads_started":1}
I20260810 14:35:56.703433 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.698863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.21.43.66:34111)
0810 14:35:56.699463 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:56.699468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:56.699481 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:56.699530 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:56.699532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:56.699576 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:56.699661 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:56.699934 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.700653 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.702131 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.702763 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.702866 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.702930 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:56.703196 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 14:35:56.703295 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":84,"threads_started":1}
I20260810 14:35:56.703701 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.698167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38713 (local address 127.0.0.1:44370)
0810 14:35:56.698255 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.698272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:56.698365 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:56.698966 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:56.698972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:56.698992 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:56.699085 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.699093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.701092 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.701099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.702287 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:56.702295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.703301 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.703304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:56.703397 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.703423 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:35:56.703500 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:35:56.703586 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":24}
I20260810 14:35:56.703995 29500 tablet_service.cc:1467] Processing CreateTablet for tablet cd89611e0ad444a4a7014b4d31620d96 (DEFAULT_TABLE table=test-table [id=7aed97cfded34460ad7d8bee765cce91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:56.704319 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd89611e0ad444a4a7014b4d31620d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:56.704547 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.698456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.21.43.67:38713)
0810 14:35:56.698711 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:56.698716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:56.698730 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:35:56.698785 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:56.698788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:56.698833 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:56.699242 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:56.699372 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.700963 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.702458 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:56.703126 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:56.704215 (+ 1089us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:56.704294 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:56.704331 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 14:35:56.704415 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260810 14:35:56.705610 29828 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: Bootstrap starting.
I20260810 14:35:56.706341 29645 tablet_service.cc:1467] Processing CreateTablet for tablet cd89611e0ad444a4a7014b4d31620d96 (DEFAULT_TABLE table=test-table [id=7aed97cfded34460ad7d8bee765cce91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:56.706634 29828 tablet_bootstrap.cc:654] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:56.706662 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd89611e0ad444a4a7014b4d31620d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:56.707055 29829 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Bootstrap starting.
I20260810 14:35:56.708134 29829 tablet_bootstrap.cc:654] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:56.708453 29828 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: No bootstrap required, opened a new log
I20260810 14:35:56.708951 29828 ts_tablet_manager.cc:1397] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:56.709484 29832 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Bootstrap starting.
I20260810 14:35:56.709506 29828 raft_consensus.cc:348] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.709633 29828 raft_consensus.cc:374] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:56.709668 29828 raft_consensus.cc:729] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efb7e93937744829323a54d7d7d1f20, State: Initialized, Role: FOLLOWER
I20260810 14:35:56.709765 29829 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: No bootstrap required, opened a new log
I20260810 14:35:56.709867 29829 ts_tablet_manager.cc:1397] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:56.709856 29828 consensus_queue.cc:260] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.710223 29828 ts_tablet_manager.cc:1428] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:56.710362 29829 raft_consensus.cc:348] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.710472 29829 raft_consensus.cc:374] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:56.710520 29829 raft_consensus.cc:729] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfc628b3d3c4c1787ff2d48c3b2a26f, State: Initialized, Role: FOLLOWER
I20260810 14:35:56.710525 29832 tablet_bootstrap.cc:654] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:56.710675 29829 consensus_queue.cc:260] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.711025 29829 ts_tablet_manager.cc:1428] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:56.711473 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.710741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:60051)
0810 14:35:56.711300 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.711364 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":120,"threads_started":1}
I20260810 14:35:56.712815 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.712363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:37139)
0810 14:35:56.712639 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.712705 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":103,"threads_started":1}
I20260810 14:35:56.712976 29423 heartbeater.cc:503] Master 127.21.43.124:46269 was elected leader, sending a full tablet report...
I20260810 14:35:56.713567 29570 heartbeater.cc:503] Master 127.21.43.124:46269 was elected leader, sending a full tablet report...
I20260810 14:35:56.715116 29832 tablet_bootstrap.cc:492] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: No bootstrap required, opened a new log
I20260810 14:35:56.715250 29832 ts_tablet_manager.cc:1397] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 14:35:56.715545 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.714002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:57295)
0810 14:35:56.714280 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.715441 (+ 1161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":123,"threads_started":1}
I20260810 14:35:56.715809 29832 raft_consensus.cc:348] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.715945 29832 raft_consensus.cc:374] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:56.715981 29832 raft_consensus.cc:729] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fabb5275c345c78f1e0d469de75525, State: Initialized, Role: FOLLOWER
I20260810 14:35:56.716140 29832 consensus_queue.cc:260] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:56.716537 29832 ts_tablet_manager.cc:1428] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:56.716483 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.713113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:47295)
0810 14:35:56.716324 (+ 3211us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.716402 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3124}
I20260810 14:35:56.718030 29725 heartbeater.cc:503] Master 127.21.43.124:46269 was elected leader, sending a full tablet report...
I20260810 14:35:56.718070 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.716836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:55985)
0810 14:35:56.717897 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.717963 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":109,"threads_started":1}
I20260810 14:35:56.719074 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.718792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:49017)
0810 14:35:56.718923 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.718993 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 14:35:56.882155 29728 tablet.cc:2369] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:56.908433 29573 tablet.cc:2369] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:56.921653 29426 tablet.cc:2369] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:57.025688 29841 raft_consensus.cc:482] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:57.025873 29841 raft_consensus.cc:504] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:57.026417 29841 leader_election.cc:288] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309), 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:57.027016 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd89611e0ad444a4a7014b4d31620d96" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efb7e93937744829323a54d7d7d1f20" is_pre_election: true
I20260810 14:35:57.027036 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd89611e0ad444a4a7014b4d31620d96" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" is_pre_election: true
I20260810 14:35:57.027237 29523 raft_consensus.cc:2454] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 0.
I20260810 14:35:57.027238 29373 raft_consensus.cc:2454] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 0.
I20260810 14:35:57.027689 29597 leader_election.cc:302] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fabb5275c345c78f1e0d469de75525, 3cfc628b3d3c4c1787ff2d48c3b2a26f; no voters:
I20260810 14:35:57.028004 29841 raft_consensus.cc:2781] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:57.028080 29841 raft_consensus.cc:482] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:57.028110 29841 raft_consensus.cc:3037] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:57.029271 29841 raft_consensus.cc:504] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:57.029666 29841 leader_election.cc:288] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 election: Requested vote from peers 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309), 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111)
I20260810 14:35:57.030189 29373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd89611e0ad444a4a7014b4d31620d96" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efb7e93937744829323a54d7d7d1f20"
I20260810 14:35:57.030316 29373 raft_consensus.cc:3037] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:57.030252 29523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd89611e0ad444a4a7014b4d31620d96" candidate_uuid: "26fabb5275c345c78f1e0d469de75525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f"
I20260810 14:35:57.030462 29523 raft_consensus.cc:3037] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:57.031677 29373 raft_consensus.cc:2454] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 1.
I20260810 14:35:57.031701 29523 raft_consensus.cc:2454] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fabb5275c345c78f1e0d469de75525 in term 1.
I20260810 14:35:57.032068 29597 leader_election.cc:302] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fabb5275c345c78f1e0d469de75525, 3cfc628b3d3c4c1787ff2d48c3b2a26f; no voters:
I20260810 14:35:57.032301 29841 raft_consensus.cc:2781] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:57.032565 29841 raft_consensus.cc:686] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 1 LEADER]: Becoming Leader. State: Replica: 26fabb5275c345c78f1e0d469de75525, State: Running, Role: LEADER
I20260810 14:35:57.032742 29841 consensus_queue.cc:237] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } }
I20260810 14:35:57.033483 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.033188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:53891)
0810 14:35:57.033315 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.033398 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:35:57.035228 29233 catalog_manager.cc:5495] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26fabb5275c345c78f1e0d469de75525 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "26fabb5275c345c78f1e0d469de75525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fabb5275c345c78f1e0d469de75525" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:57.069248 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.066290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.21.43.67:38713)
0810 14:35:57.066435 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:57.066439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:57.066474 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:35:57.066522 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:57.066524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:57.066571 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:57.066628 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:57.066874 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:57.067420 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:57.068254 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:57.068464 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:57.068541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:57.068780 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:57.068855 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:57.068908 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:57.068992 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 14:35:57.069042 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:35:57.071732 29523 raft_consensus.cc:1261] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Refusing update from remote peer 26fabb5275c345c78f1e0d469de75525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:57.071730 29373 raft_consensus.cc:1261] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Refusing update from remote peer 26fabb5275c345c78f1e0d469de75525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:57.072278 29843 consensus_queue.cc:1035] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfc628b3d3c4c1787ff2d48c3b2a26f" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:57.072465 29841 consensus_queue.cc:1035] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efb7e93937744829323a54d7d7d1f20" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:57.075736 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.075401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:59557)
0810 14:35:57.075563 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.075633 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:35:57.076371 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.076129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:44411)
0810 14:35:57.076252 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.076304 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:35:57.079293 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.079013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.65:38149)
0810 14:35:57.079118 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.079209 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 14:35:57.080978 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.079264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:47285)
0810 14:35:57.080766 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.080864 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1393}
I20260810 14:35:57.082952 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.082580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.67:44173)
0810 14:35:57.082783 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.082847 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 14:35:57.084954 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.083373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34473 (local address 127.21.43.66:36149)
0810 14:35:57.084796 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.084861 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1335}
I20260810 14:35:57.085530 29232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:57.085698 29232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:57.090538 29232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f1c7c9c1f24452a8515d5951b091680: Sending DeleteTablet for 3 replicas of tablet cd89611e0ad444a4a7014b4d31620d96
I20260810 14:35:57.091431 29645 tablet_service.cc:1514] Processing DeleteTablet for tablet cd89611e0ad444a4a7014b4d31620d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:57 UTC) from {username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:44370
I20260810 14:35:57.091619 29352 tablet_service.cc:1514] Processing DeleteTablet for tablet cd89611e0ad444a4a7014b4d31620d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:57 UTC) from {username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:45362
I20260810 14:35:57.091878 29500 tablet_service.cc:1514] Processing DeleteTablet for tablet cd89611e0ad444a4a7014b4d31620d96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:57 UTC) from {username='oryx', principal='oryx/127.21.43.124@KRBTEST.COM'} at 127.0.0.1:39100
I20260810 14:35:57.092172 29855 tablet_replica.cc:330] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: stopping tablet replica
I20260810 14:35:57.092288 29858 tablet_replica.cc:330] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20: stopping tablet replica
I20260810 14:35:57.092362 29855 raft_consensus.cc:2229] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:57.092410 29858 raft_consensus.cc:2229] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:57.092562 29858 raft_consensus.cc:2258] T cd89611e0ad444a4a7014b4d31620d96 P 8efb7e93937744829323a54d7d7d1f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:57.092600 29855 raft_consensus.cc:2258] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:57.092968 29859 tablet_replica.cc:330] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: stopping tablet replica
I20260810 14:35:57.093101 29859 raft_consensus.cc:2229] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:57.093240 29859 raft_consensus.cc:2258] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:57.093295 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29195
I20260810 14:35:57.096789 29859 ts_tablet_manager.cc:1905] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:57.097134 29855 ts_tablet_manager.cc:1905] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:57.099356 29859 ts_tablet_manager.cc:1918] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:57.099470 29859 log.cc:1196] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/cd89611e0ad444a4a7014b4d31620d96
I20260810 14:35:57.099750 29859 ts_tablet_manager.cc:1939] T cd89611e0ad444a4a7014b4d31620d96 P 3cfc628b3d3c4c1787ff2d48c3b2a26f: Deleting consensus metadata
I20260810 14:35:57.100611 29219 catalog_manager.cc:4841] TS 3cfc628b3d3c4c1787ff2d48c3b2a26f (127.21.43.66:34111): tablet cd89611e0ad444a4a7014b4d31620d96 (table test-table [id=7aed97cfded34460ad7d8bee765cce91]) successfully deleted
I20260810 14:35:57.100862 29855 ts_tablet_manager.cc:1918] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:57.100947 29855 log.cc:1196] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/cd89611e0ad444a4a7014b4d31620d96
I20260810 14:35:57.101285 29855 ts_tablet_manager.cc:1939] T cd89611e0ad444a4a7014b4d31620d96 P 26fabb5275c345c78f1e0d469de75525: Deleting consensus metadata
I20260810 14:35:57.102252 29217 catalog_manager.cc:4841] TS 26fabb5275c345c78f1e0d469de75525 (127.21.43.67:38713): tablet cd89611e0ad444a4a7014b4d31620d96 (table test-table [id=7aed97cfded34460ad7d8bee765cce91]) successfully deleted
I20260810 14:35:57.104563 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29431
I20260810 14:35:57.105033 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.104653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42309 (local address 127.0.0.1:45376)
0810 14:35:57.104875 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.104932 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260810 14:35:57.108204 29220 catalog_manager.cc:4584] TS 8efb7e93937744829323a54d7d7d1f20 (127.21.43.65:42309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd89611e0ad444a4a7014b4d31620d96: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42309: connect: Connection refused (error 111)
I20260810 14:35:57.115917 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29577
I20260810 14:35:57.127986 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 28892
I20260810 14:35:57.138968 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29002
2026-08-10T14:35:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5896 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:57 dist-test-slave-chsk krb5kdc[29862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:57 dist-test-slave-chsk krb5kdc[29862](info): set up 2 sockets
Aug 10 14:35:57 dist-test-slave-chsk krb5kdc[29862](info): commencing operation
krb5kdc: starting...
W20260810 14:35:58.243620 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:35:58 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab.
Entry for principal oryx/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab.
I20260810 14:35:58.379798 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38427
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab
--principal=oryx/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:58.494858 29875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:58.495216 29875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:58.495326 29875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:58.499526 29875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:58.499624 29875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:35:58.499657 29875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:58.499701 29875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:58.499739 29875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:58.499775 29875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:58.505271 29875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38427
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.126.keytab
--principal=oryx/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:58.506481 29875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:58.507558 29875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:58.515957 29880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:58.516031 29883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:58.515957 29881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:58.516371 29875 server_base.cc:1034] running on GCE node
I20260810 14:35:58.516943 29875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:58.518302 29875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:58.519507 29875 hybrid_clock.cc:648] HybridClock initialized: now 1786372558519488 us; error 53 us; skew 500 ppm
Aug 10 14:35:58 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:58.522297 29875 init.cc:377] Logged in from keytab as oryx/127.21.43.126@KRBTEST.COM (short username oryx)
I20260810 14:35:58.523293 29875 webserver.cc:460] Webserver started at http://127.21.43.126:43647/ using document root <none> and password file <none>
I20260810 14:35:58.523864 29875 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:58.523957 29875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:58.524204 29875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:58.526274 29875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "414209e6550442909a9fa008d7645a67"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "fe731726bdf4c966a541d8fb36ca01eb"
server_key_iv: "ec30ccef20e3c9185c504791d4ce1c3b"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.526867 29875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "414209e6550442909a9fa008d7645a67"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "fe731726bdf4c966a541d8fb36ca01eb"
server_key_iv: "ec30ccef20e3c9185c504791d4ce1c3b"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.530730 29875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 14:35:58.533329 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:58.534657 29875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:58.534849 29875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "414209e6550442909a9fa008d7645a67"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "fe731726bdf4c966a541d8fb36ca01eb"
server_key_iv: "ec30ccef20e3c9185c504791d4ce1c3b"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.534979 29875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:58.552205 29875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:58.555518 29875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:58.555753 29875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:58.563843 29875 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38427
I20260810 14:35:58.563874 29958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38427 every 8 connection(s)
I20260810 14:35:58.564831 29875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:58.567502 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29875
I20260810 14:35:58.567638 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:35:58.567780 29959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:58.567966 21677 external_mini_cluster.cc:1428] Setting key d4593d0c97dee34c8f6bf2d11ce02bc1
I20260810 14:35:58.575253 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Bootstrap starting.
Aug 10 14:35:58 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:58.577972 29959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:58.579427 29959 log.cc:824] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:58.581914 29959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: No bootstrap required, opened a new log
I20260810 14:35:58.583423 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.569421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.21.43.126:38427)
0810 14:35:58.569869 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:58.569888 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:58.569932 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:35:58.570595 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:58.570618 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:58.571058 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:58.571448 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:58.572124 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.573583 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.574410 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.575081 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.575285 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:58.578030 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:58.578037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:58.578053 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:58.578086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:58.580060 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.580566 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.580572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.580578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.580716 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.580986 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.580989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.580991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.581518 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:58.581699 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:58.581941 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 14:35:58.582460 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260810 14:35:58.585853 29959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } }
I20260810 14:35:58.586074 29959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:58.586124 29959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414209e6550442909a9fa008d7645a67, State: Initialized, Role: FOLLOWER
I20260810 14:35:58.586539 29959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } }
I20260810 14:35:58.586668 29959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:58.586715 29959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:58.586786 29959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:58.587790 29959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } }
I20260810 14:35:58.588132 29959 leader_election.cc:302] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 414209e6550442909a9fa008d7645a67; no voters:
I20260810 14:35:58.588532 29959 leader_election.cc:288] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:58.588762 29964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:58.588965 29964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [term 1 LEADER]: Becoming Leader. State: Replica: 414209e6550442909a9fa008d7645a67, State: Running, Role: LEADER
I20260810 14:35:58.589270 29964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } }
I20260810 14:35:58.589895 29959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:58.591082 29965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "414209e6550442909a9fa008d7645a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } } }
I20260810 14:35:58.591519 29965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:58.591993 29975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:58.591959 29966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 414209e6550442909a9fa008d7645a67. Latest consensus state: current_term: 1 leader_uuid: "414209e6550442909a9fa008d7645a67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414209e6550442909a9fa008d7645a67" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38427 } } }
I20260810 14:35:58.592173 29966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:58.594924 29975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:58.601506 29975 catalog_manager.cc:1347] Generated new cluster ID: 0f05d3ae101a45d4bd3cf6f737062005
I20260810 14:35:58.601599 29975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:58.613919 29975 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:58.614953 29975 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:58.624166 29975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Generated new TSK 0
I20260810 14:35:58.624923 29975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
I20260810 14:35:58.658833 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38427
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:58.806104 29987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:58.806483 29987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:58.806538 29987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:58.810585 29987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:58.810650 29987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:58.810693 29987 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:58.810729 29987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:35:58.816108 29987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:58.817332 29987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:58.818364 29987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:58.828394 29992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:58.828384 29993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:58.828403 29995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:58.828598 29987 server_base.cc:1034] running on GCE node
I20260810 14:35:58.829010 29987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:58.829696 29987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:58.830904 29987 hybrid_clock.cc:648] HybridClock initialized: now 1786372558830890 us; error 35 us; skew 500 ppm
Aug 10 14:35:58 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:58.833871 29987 init.cc:377] Logged in from keytab as oryx/127.21.43.65@KRBTEST.COM (short username oryx)
I20260810 14:35:58.834848 29987 webserver.cc:460] Webserver started at http://127.21.43.65:44867/ using document root <none> and password file <none>
I20260810 14:35:58.835502 29987 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:58.835597 29987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:58.835850 29987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:58.838078 29987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "fca47d519173443d972c7783d92aaf9b"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "f70b724ce04b8932e6ef38f602d0da13"
server_key_iv: "59782fb4a6379f24c1f5d3f2e3c2c37c"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.838691 29987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "fca47d519173443d972c7783d92aaf9b"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "f70b724ce04b8932e6ef38f602d0da13"
server_key_iv: "59782fb4a6379f24c1f5d3f2e3c2c37c"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.842674 29987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 14:35:58.845299 30003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:58.846554 29987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:58.846701 29987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "fca47d519173443d972c7783d92aaf9b"
format_stamp: "Formatted at 2026-08-10 14:35:58 on dist-test-slave-chsk"
server_key: "f70b724ce04b8932e6ef38f602d0da13"
server_key_iv: "59782fb4a6379f24c1f5d3f2e3c2c37c"
server_key_version: "encryptionkey@0"
I20260810 14:35:58.846834 29987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:58.867776 29987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:58.870831 29987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:58.871002 29987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:58.872691 29987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:58.872761 29987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:58.872805 29987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:58.872864 29987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:58 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:58.884238 29987 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39727
I20260810 14:35:58.885008 30124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39727 every 8 connection(s)
I20260810 14:35:58.885216 29987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:58.886993 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.873475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:52263 (local address 127.21.43.126:38427)
0810 14:35:58.873669 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:58.873677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:58.874526 (+ 849us) server_negotiation.cc:407] Connection header received
0810 14:35:58.875728 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:58.875731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:58.875794 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:58.875893 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:58.877617 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.878345 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.879761 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.880081 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.880204 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:58.883180 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:58.883184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:58.883186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:58.883220 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:58.884786 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.885399 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.885403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.885405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.885492 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.885927 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.885930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.885932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.886181 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:58.886265 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:58.886762 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 14:35:58.886870 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:35:58.887817 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 29987
I20260810 14:35:58.887821 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.873859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.65:52263)
0810 14:35:58.874346 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:58.874397 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:58.875456 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:58.876127 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:58.876140 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:58.876684 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:58.877447 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:58.877473 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.878514 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.878518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:58.879621 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:58.879632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.880269 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.880274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:58.880439 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:58.881096 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:58.881115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:58.882971 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:58.884912 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:58.884918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:58.884935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:58.885286 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:58.885626 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:58.885630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:58.885631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:58.885802 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:58.886294 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:58.886301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:58.886549 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 14:35:58.886714 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 14:35:58.886977 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
I20260810 14:35:58.887952 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:58.888258 21677 external_mini_cluster.cc:1428] Setting key dd215866ca61a318ccc512dc28faf039
I20260810 14:35:58.889309 30125 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38427
I20260810 14:35:58.889490 30125 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:58.890115 30125 heartbeater.cc:511] Master 127.21.43.126:38427 requested a full tablet report, sending...
I20260810 14:35:58.891937 29908 ts_manager.cc:194] Registered new tserver with Master: fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727)
I20260810 14:35:58.893321 29908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:52263
WARNING: no policy specified for oryx/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:35:58.900573 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.891737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.65:44711)
0810 14:35:58.892153 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:58.892192 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:58.892367 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:58.892678 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:58.892683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:58.892945 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:58.893102 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:58.893111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.894135 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.894138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:58.895066 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:58.895078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.895951 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.895955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:58.896064 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:58.896658 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:58.896670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:58.897037 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:58.898980 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:58.898982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:58.898984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:58.899275 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:58.899580 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:58.899582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:58.899584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:58.899662 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:58.900145 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:58.900147 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:58.900222 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 14:35:58.900300 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:35:58.900411 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":19,"thread_start_us":240,"threads_started":1}
I20260810 14:35:58.900729 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.891794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:44711 (local address 127.21.43.126:38427)
0810 14:35:58.891907 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:58.891912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:58.892284 (+ 372us) server_negotiation.cc:407] Connection header received
0810 14:35:58.892506 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:58.892509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:58.892557 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:58.892642 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:58.893243 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.893975 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.895233 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:58.895800 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:58.895930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:58.897198 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:58.897204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:58.897212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:58.897246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:58.898863 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.899380 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.899384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.899386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.899472 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:58.899756 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:58.899761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:58.899763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:58.900035 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:58.900150 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:58.900386 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 14:35:58.900513 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
I20260810 14:35:58.920742 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38427
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:59.035636 30136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:59.035916 30136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:59.035964 30136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:59.040386 30136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:59.040455 30136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:59.040516 30136 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:59.040584 30136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:35:59.046109 30136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:59.047297 30136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:59.048346 30136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:59.058182 30144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:59.058177 30142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:59.058311 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:59.058600 30136 server_base.cc:1034] running on GCE node
I20260810 14:35:59.059006 30136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:59.059691 30136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:59.060856 30136 hybrid_clock.cc:648] HybridClock initialized: now 1786372559060837 us; error 33 us; skew 500 ppm
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:59.063736 30136 init.cc:377] Logged in from keytab as oryx/127.21.43.66@KRBTEST.COM (short username oryx)
I20260810 14:35:59.064640 30136 webserver.cc:460] Webserver started at http://127.21.43.66:38805/ using document root <none> and password file <none>
I20260810 14:35:59.065236 30136 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:59.065302 30136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:59.065523 30136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:59.067786 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "e64e413128bc45748e8f8e507eb48d58"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "f288ed5f4f5d5df9a9fecd7b276f6360"
server_key_iv: "0e74fd7a9e1e96b9325af014ee804a56"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.068372 30136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "e64e413128bc45748e8f8e507eb48d58"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "f288ed5f4f5d5df9a9fecd7b276f6360"
server_key_iv: "0e74fd7a9e1e96b9325af014ee804a56"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.072223 30136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:59.074736 30151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:59.076074 30136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:59.076205 30136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "e64e413128bc45748e8f8e507eb48d58"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "f288ed5f4f5d5df9a9fecd7b276f6360"
server_key_iv: "0e74fd7a9e1e96b9325af014ee804a56"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.076344 30136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:59.086687 30136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:59.089784 30136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:59.090039 30136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:59.091950 30136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:59.092002 30136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:59.092075 30136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:59.092109 30136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:59.103678 30136 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:46699
I20260810 14:35:59.103766 30272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:46699 every 8 connection(s)
I20260810 14:35:59.104588 30136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:59.107064 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.094303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:40087 (local address 127.21.43.126:38427)
0810 14:35:59.094465 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.094470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.094488 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:35:59.094807 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.094810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.094868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.094969 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.096577 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.097449 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.099367 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.099739 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.099881 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.103051 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.103055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.103058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.103087 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.104909 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.105468 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.105472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.105473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.105565 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.105919 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.105923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.105925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.106186 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.106284 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.106761 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.106892 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:35:59.107827 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.093186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.66:40087)
0810 14:35:59.093702 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.093745 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.094601 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.095186 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.095194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.095721 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:59.096435 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.096457 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.097598 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.097604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.099216 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.099242 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.099956 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.099964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.100208 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.100854 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:59.100878 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:59.102874 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:59.105042 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.105047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.105060 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.105377 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.105661 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.105665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.105667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.105819 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.106284 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:59.106292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:59.106525 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.106714 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.107016 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":144,"threads_started":1}
I20260810 14:35:59.108137 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 30136
I20260810 14:35:59.108263 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:59.108634 21677 external_mini_cluster.cc:1428] Setting key d8a2c775657777d383d4e7510d45494a
I20260810 14:35:59.109205 30273 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38427
I20260810 14:35:59.109342 30273 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:59.109936 30273 heartbeater.cc:511] Master 127.21.43.126:38427 requested a full tablet report, sending...
I20260810 14:35:59.111025 29909 ts_manager.cc:194] Registered new tserver with Master: e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:35:59.111804 29909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:40087
WARNING: no policy specified for oryx/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:35:59.119871 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.111510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:41957 (local address 127.21.43.126:38427)
0810 14:35:59.111623 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.111627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.111923 (+ 296us) server_negotiation.cc:407] Connection header received
0810 14:35:59.112291 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.112296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.112347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.112399 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.113026 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.113760 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.114645 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.114861 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.114972 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.116136 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.116141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.116150 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.116171 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.117599 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.118141 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.118145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.118147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.118257 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.118652 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.118656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.118658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.119012 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.119118 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.119557 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.119688 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:35:59.119897 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.111461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.66:41957)
0810 14:35:59.111815 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.111833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.112002 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.112532 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.112535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.112761 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:59.112886 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.112893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.113923 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.113926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.114506 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.114515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.114979 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.114983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.115071 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.115680 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:59.115692 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:59.115996 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:59.117754 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.117758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.117760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.118010 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.118389 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.118392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.118395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.118496 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.119172 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:59.119176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:59.119310 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.119474 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.119664 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
I20260810 14:35:59.138461 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38427
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:59.251463 30284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:59.251720 30284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:59.251797 30284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:59.255733 30284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:59.255780 30284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:59.255853 30284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:35:59.255937 30284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:35:59.261039 30284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.43.126:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:35:59.262032 30284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:59.263020 30284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:59.272542 30289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:59.272723 30290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:59.272847 30292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:59.272787 30284 server_base.cc:1034] running on GCE node
I20260810 14:35:59.273277 30284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:59.273880 30284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:59.275070 30284 hybrid_clock.cc:648] HybridClock initialized: now 1786372559275059 us; error 25 us; skew 500 ppm
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:59.277798 30284 init.cc:377] Logged in from keytab as oryx/127.21.43.67@KRBTEST.COM (short username oryx)
I20260810 14:35:59.278630 30284 webserver.cc:460] Webserver started at http://127.21.43.67:35397/ using document root <none> and password file <none>
I20260810 14:35:59.279250 30284 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:59.279347 30284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:59.279624 30284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:59.281689 30284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "3f43453585554dcba1cf4d40492d231c"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "a6f59e52a80a7d0b9762eff8d4c9d70e"
server_key_iv: "b60b39b07b53b3ad4ab6f4bbb3cc312c"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.282274 30284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "3f43453585554dcba1cf4d40492d231c"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "a6f59e52a80a7d0b9762eff8d4c9d70e"
server_key_iv: "b60b39b07b53b3ad4ab6f4bbb3cc312c"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.286110 30284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:35:59.288691 30299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:59.290092 30284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:59.290212 30284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "3f43453585554dcba1cf4d40492d231c"
format_stamp: "Formatted at 2026-08-10 14:35:59 on dist-test-slave-chsk"
server_key: "a6f59e52a80a7d0b9762eff8d4c9d70e"
server_key_iv: "b60b39b07b53b3ad4ab6f4bbb3cc312c"
server_key_version: "encryptionkey@0"
I20260810 14:35:59.290413 30284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:59.313567 30284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:59.316816 30284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:59.316996 30284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:59.318446 30284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:59.318529 30284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:59.318601 30284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:59.318651 30284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:59.330521 30284 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:43091
I20260810 14:35:59.330596 30426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:43091 every 8 connection(s)
I20260810 14:35:59.331632 30284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:59.334131 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.319410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:52891 (local address 127.21.43.126:38427)
0810 14:35:59.319553 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.319556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.320681 (+ 1125us) server_negotiation.cc:407] Connection header received
0810 14:35:59.321680 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.321684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.321745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.321808 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.323787 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.324784 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.326476 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.326903 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.327048 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.330216 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.330220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.330223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.330258 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.331817 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.332457 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.332461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.332462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.332554 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.333018 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.333022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.333023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.333249 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.333344 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.333855 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.333971 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:59.334968 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.319964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.67:52891)
0810 14:35:59.320510 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.320552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.321411 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.322336 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.322347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.322856 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:59.323617 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.323640 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.324936 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.324941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.326335 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.326347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.327124 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.327129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.327351 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.328031 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:59.328047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:59.329997 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:59.331939 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.331945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.331959 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.332328 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.332696 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.332699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.332701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.332872 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.333386 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:59.333392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:59.333623 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.333782 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.334086 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":146,"threads_started":1}
I20260810 14:35:59.335461 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 30284
I20260810 14:35:59.335641 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:59.336032 21677 external_mini_cluster.cc:1428] Setting key 8cdfb47882205721bd48c5d2fee3fd24
I20260810 14:35:59.336517 30428 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38427
I20260810 14:35:59.336700 30428 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:59.337556 30428 heartbeater.cc:511] Master 127.21.43.126:38427 requested a full tablet report, sending...
I20260810 14:35:59.338619 29908 ts_manager.cc:194] Registered new tserver with Master: 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091)
I20260810 14:35:59.339522 29908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:52891
I20260810 14:35:59.346238 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.338829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:49939 (local address 127.21.43.126:38427)
0810 14:35:59.338970 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.338974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.339230 (+ 256us) server_negotiation.cc:407] Connection header received
0810 14:35:59.339350 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.339352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.339392 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.339499 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.340004 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.340536 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.341472 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.341733 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.341854 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.343110 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.343114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.343117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.343169 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.344529 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.344988 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.344992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.344994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.345099 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.345389 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.345393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.345395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.345579 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.345691 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.345955 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.346079 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:35:59.346247 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.338802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.21.43.67:49939)
0810 14:35:59.339134 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.339171 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.339270 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.339534 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.339540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.339759 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:59.339884 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.339892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.340687 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.340690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.341338 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.341347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.341858 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.341862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.341967 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.342571 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:59.342584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:59.342948 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:59.344663 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.344668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.344670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.344880 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.345198 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.345201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.345203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.345291 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.345682 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:59.345684 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:59.345786 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.345898 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.346053 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":132,"threads_started":1}
I20260810 14:35:59.350026 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:35:59.375948 29962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:35:59.369390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.21.43.126:38427)
0810 14:35:59.369525 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.369529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.369542 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:59.369587 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.369590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.369631 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.369704 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.370474 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.371039 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.372022 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.372252 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.372339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.375822 (+ 3483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57964: BlockingRecv error: recv got EOF from 127.0.0.1:57964 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372559, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:59.386042 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.377590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.21.43.126:38427)
0810 14:35:59.377751 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.377755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.377881 (+ 126us) server_negotiation.cc:407] Connection header received
0810 14:35:59.378048 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.378052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.378106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.378191 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.378797 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.379391 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.380286 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.380501 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.380625 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.383021 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.383026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.383028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.383059 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.384385 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.384800 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.384804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.384806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.384889 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.385178 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.385183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.385184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.385376 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.385520 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.385752 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.385871 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:35:59.389118 29908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:59.392720 29908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:59.407737 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.401541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39727 (local address 127.0.0.1:41926)
0810 14:35:59.402527 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.402582 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.402782 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.403415 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.403424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.403455 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.403603 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.403610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.404928 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.404933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.406171 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.406181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.407098 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.407102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.407242 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.407300 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.407407 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.407548 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":76,"threads_started":1}
I20260810 14:35:59.407853 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.401706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.21.43.65:39727)
0810 14:35:59.402017 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.402024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.402669 (+ 645us) server_negotiation.cc:407] Connection header received
0810 14:35:59.402990 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.402999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.403249 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.403409 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.403759 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.404815 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.406306 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.406973 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.407094 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.407362 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.407475 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.407608 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
I20260810 14:35:59.408288 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.401541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:43091 (local address 127.0.0.1:58438)
0810 14:35:59.403249 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.403265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.403369 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.403948 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.403953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.403976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.404105 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.404110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.405371 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.405374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.406527 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.406537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.407801 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.407804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.407915 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.407943 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.408025 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.408103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1625,"mutex_wait_us":611,"thread_start_us":100,"threads_started":1}
I20260810 14:35:59.408756 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.401771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.21.43.67:43091)
0810 14:35:59.402252 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.402266 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.403387 (+ 1121us) server_negotiation.cc:407] Connection header received
0810 14:35:59.403484 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.403488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.403670 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.403808 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.404223 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.405233 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.406670 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.407474 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.407584 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.407734 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.408475 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.408618 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":136,"threads_started":1}
I20260810 14:35:59.409587 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.402846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.21.43.66:46699)
0810 14:35:59.403243 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.403252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.403844 (+ 592us) server_negotiation.cc:407] Connection header received
0810 14:35:59.403921 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.403926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.404096 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.404247 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.404635 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.405743 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.406990 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.408981 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.409116 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.409278 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.409369 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.409451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
I20260810 14:35:59.409559 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.402524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.0.0.1:57498)
0810 14:35:59.403625 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.403638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.403748 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.404380 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.404383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.404400 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.404507 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.404513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.405870 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.405873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.406878 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.406885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.409117 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.409121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.409220 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.409242 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.409360 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.409435 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":103,"threads_started":1}
I20260810 14:35:59.411469 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 292e794e507f41b3b806c7732231bf94 (DEFAULT_TABLE table=test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.411665 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 292e794e507f41b3b806c7732231bf94 (DEFAULT_TABLE table=test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.412602 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292e794e507f41b3b806c7732231bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.412632 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292e794e507f41b3b806c7732231bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.413995 30200 tablet_service.cc:1467] Processing CreateTablet for tablet 292e794e507f41b3b806c7732231bf94 (DEFAULT_TABLE table=test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.415004 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292e794e507f41b3b806c7732231bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.419600 30459 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Bootstrap starting.
I20260810 14:35:59.420667 30460 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Bootstrap starting.
I20260810 14:35:59.421692 30459 tablet_bootstrap.cc:654] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.422518 30459 log.cc:824] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:59.422674 30461 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Bootstrap starting.
I20260810 14:35:59.422762 30460 tablet_bootstrap.cc:654] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.423666 30460 log.cc:824] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:59.424988 30461 tablet_bootstrap.cc:654] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.425249 30459 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: No bootstrap required, opened a new log
I20260810 14:35:59.425451 30459 ts_tablet_manager.cc:1397] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:35:59.425747 30460 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: No bootstrap required, opened a new log
I20260810 14:35:59.425825 30461 log.cc:824] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:59.425951 30460 ts_tablet_manager.cc:1397] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:35:59.427837 30461 tablet_bootstrap.cc:492] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: No bootstrap required, opened a new log
I20260810 14:35:59.428027 30461 ts_tablet_manager.cc:1397] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260810 14:35:59.429556 30460 raft_consensus.cc:348] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.429582 30459 raft_consensus.cc:348] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.429808 30460 raft_consensus.cc:374] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.429828 30459 raft_consensus.cc:374] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.429863 30460 raft_consensus.cc:729] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.429880 30459 raft_consensus.cc:729] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.430311 30460 consensus_queue.cc:260] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.430320 30459 consensus_queue.cc:260] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.431337 30125 heartbeater.cc:503] Master 127.21.43.126:38427 was elected leader, sending a full tablet report...
I20260810 14:35:59.431381 30459 ts_tablet_manager.cc:1428] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 14:35:59.431710 30428 heartbeater.cc:503] Master 127.21.43.126:38427 was elected leader, sending a full tablet report...
I20260810 14:35:59.432052 30460 ts_tablet_manager.cc:1428] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260810 14:35:59.432436 30461 raft_consensus.cc:348] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.432782 30461 raft_consensus.cc:374] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.432875 30461 raft_consensus.cc:729] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.433300 30461 consensus_queue.cc:260] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.433987 30273 heartbeater.cc:503] Master 127.21.43.126:38427 was elected leader, sending a full tablet report...
I20260810 14:35:59.434317 30461 ts_tablet_manager.cc:1428] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 14:35:59.440169 30465 raft_consensus.cc:482] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:59.440392 30465 raft_consensus.cc:504] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.441511 30465 leader_election.cc:288] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:35:59.446476 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.442220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:52083 (local address 127.21.43.66:46699)
0810 14:35:59.442352 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.442357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.442585 (+ 228us) server_negotiation.cc:407] Connection header received
0810 14:35:59.442669 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.442671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.442733 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.442814 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.443034 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.443775 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445193 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.445816 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445949 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.446014 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.446279 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.446351 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:35:59.446476 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.441801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:38213 (local address 127.21.43.67:43091)
0810 14:35:59.441930 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.441934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.442103 (+ 169us) server_negotiation.cc:407] Connection header received
0810 14:35:59.442293 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.442298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.442352 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.442436 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.442748 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.443591 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445193 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.445816 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445949 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.446002 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.446282 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.446351 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:35:59.446537 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.441683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:43091 (local address 127.21.43.65:38213)
0810 14:35:59.441969 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.441999 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.442123 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.442465 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.442469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.442489 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.442632 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.442642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.443709 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.443711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.445040 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.445047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.445950 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.446120 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.446171 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.446247 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.446341 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":95,"thread_start_us":132,"threads_started":1}
I20260810 14:35:59.446538 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.442121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.21.43.65:52083)
0810 14:35:59.442481 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.442493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.442596 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.442835 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.442837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.442851 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.442934 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.442938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.443868 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.443871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.445036 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.445047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.445950 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.445954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.446092 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.446168 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.446247 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.446341 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":58,"thread_start_us":98,"threads_started":1}
I20260810 14:35:59.447201 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292e794e507f41b3b806c7732231bf94" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c" is_pre_election: true
I20260810 14:35:59.447249 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292e794e507f41b3b806c7732231bf94" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58" is_pre_election: true
I20260810 14:35:59.447527 30377 raft_consensus.cc:2454] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 0.
I20260810 14:35:59.447535 30224 raft_consensus.cc:2454] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 0.
I20260810 14:35:59.448014 30006 leader_election.cc:302] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64e413128bc45748e8f8e507eb48d58, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:35:59.448289 30465 raft_consensus.cc:2781] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:59.448369 30465 raft_consensus.cc:482] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:59.448400 30465 raft_consensus.cc:3037] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.449415 30465 raft_consensus.cc:504] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.449764 30465 leader_election.cc:288] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 election: Requested vote from peers 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:35:59.450137 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292e794e507f41b3b806c7732231bf94" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c"
I20260810 14:35:59.450261 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292e794e507f41b3b806c7732231bf94" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58"
I20260810 14:35:59.450265 30377 raft_consensus.cc:3037] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.450392 30224 raft_consensus.cc:3037] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.451556 30224 raft_consensus.cc:2454] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 1.
I20260810 14:35:59.451686 30377 raft_consensus.cc:2454] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 1.
I20260810 14:35:59.451915 30006 leader_election.cc:302] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64e413128bc45748e8f8e507eb48d58, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:35:59.452096 30465 raft_consensus.cc:2781] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:59.452329 30465 raft_consensus.cc:686] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 1 LEADER]: Becoming Leader. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Running, Role: LEADER
I20260810 14:35:59.452692 30465 consensus_queue.cc:237] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } }
I20260810 14:35:59.455847 29909 catalog_manager.cc:5495] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b reported cstate change: term changed from 0 to 1, leader changed from <none> to fca47d519173443d972c7783d92aaf9b (127.21.43.65). New cstate: current_term: 1 leader_uuid: "fca47d519173443d972c7783d92aaf9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:59.471696 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.468097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.21.43.65:39727)
0810 14:35:59.468234 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.468237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.468250 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:35:59.468325 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.468327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.468381 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.468477 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:59.468731 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.469394 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.470231 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.470454 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.470544 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.470784 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.471280 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.471374 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.471517 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.471583 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:35:59.477911 30224 raft_consensus.cc:1261] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Refusing update from remote peer fca47d519173443d972c7783d92aaf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.477900 30377 raft_consensus.cc:1261] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Refusing update from remote peer fca47d519173443d972c7783d92aaf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.478741 30465 consensus_queue.cc:1035] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.478996 30470 consensus_queue.cc:1035] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.487818 30473 mvcc.cc:205] Tried to move back new op lower bound from 7316982003613945856 to 7316982003521220608. Current Snapshot: MvccSnapshot[applied={T|T < 7316982003613945856}]
I20260810 14:35:59.489097 30475 mvcc.cc:205] Tried to move back new op lower bound from 7316982003613945856 to 7316982003521220608. Current Snapshot: MvccSnapshot[applied={T|T < 7316982003613945856}]
I20260810 14:35:59.490574 30474 mvcc.cc:205] Tried to move back new op lower bound from 7316982003613945856 to 7316982003521220608. Current Snapshot: MvccSnapshot[applied={T|T < 7316982003613945856}]
I20260810 14:35:59.499033 29909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:59.499264 29909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:59.501967 29909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Sending DeleteTablet for 3 replicas of tablet 292e794e507f41b3b806c7732231bf94
I20260810 14:35:59.502800 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet 292e794e507f41b3b806c7732231bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41926
I20260810 14:35:59.502914 30200 tablet_service.cc:1514] Processing DeleteTablet for tablet 292e794e507f41b3b806c7732231bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57498
I20260810 14:35:59.502964 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 292e794e507f41b3b806c7732231bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:58438
I20260810 14:35:59.503573 30487 tablet_replica.cc:330] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: stopping tablet replica
I20260810 14:35:59.503625 30485 tablet_replica.cc:330] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: stopping tablet replica
I20260810 14:35:59.503930 30485 raft_consensus.cc:2229] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:59.503930 30487 raft_consensus.cc:2229] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:59.504303 30487 raft_consensus.cc:2258] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.504441 30485 raft_consensus.cc:2258] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.504398 29909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:59.504859 29909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:59.506031 30487 ts_tablet_manager.cc:1905] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:59.506448 30485 ts_tablet_manager.cc:1905] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:59.509289 30487 ts_tablet_manager.cc:1918] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:59.509441 30487 log.cc:1196] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/292e794e507f41b3b806c7732231bf94
I20260810 14:35:59.509788 30487 ts_tablet_manager.cc:1939] T 292e794e507f41b3b806c7732231bf94 P 3f43453585554dcba1cf4d40492d231c: Deleting consensus metadata
I20260810 14:35:59.510110 30056 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9291897940feb96cd7422b09e5c5 (DEFAULT_TABLE table=test-table [id=49e565d8f1264d69ba6544ef58cddb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.510586 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a9291897940feb96cd7422b09e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.510614 30200 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9291897940feb96cd7422b09e5c5 (DEFAULT_TABLE table=test-table [id=49e565d8f1264d69ba6544ef58cddb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.510630 30352 tablet_service.cc:1467] Processing CreateTablet for tablet f83a9291897940feb96cd7422b09e5c5 (DEFAULT_TABLE table=test-table [id=49e565d8f1264d69ba6544ef58cddb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:59.510883 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a9291897940feb96cd7422b09e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.510905 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83a9291897940feb96cd7422b09e5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.513572 30460 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Bootstrap starting.
I20260810 14:35:59.513747 30461 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Bootstrap starting.
I20260810 14:35:59.514654 30460 tablet_bootstrap.cc:654] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.515475 30459 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Bootstrap starting.
I20260810 14:35:59.515677 30486 tablet_replica.cc:330] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: stopping tablet replica
I20260810 14:35:59.516265 30485 ts_tablet_manager.cc:1918] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:59.516376 30485 log.cc:1196] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/292e794e507f41b3b806c7732231bf94
I20260810 14:35:59.516574 30459 tablet_bootstrap.cc:654] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.516752 30485 ts_tablet_manager.cc:1939] T 292e794e507f41b3b806c7732231bf94 P fca47d519173443d972c7783d92aaf9b: Deleting consensus metadata
I20260810 14:35:59.517735 30461 tablet_bootstrap.cc:654] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.518253 30459 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: No bootstrap required, opened a new log
I20260810 14:35:59.518350 30459 ts_tablet_manager.cc:1397] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:59.518713 29895 catalog_manager.cc:4841] TS fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727): tablet 292e794e507f41b3b806c7732231bf94 (table test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]) successfully deleted
I20260810 14:35:59.518803 30459 raft_consensus.cc:348] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.518875 30486 raft_consensus.cc:2229] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:59.518932 30459 raft_consensus.cc:374] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.518972 30459 raft_consensus.cc:729] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.519266 30486 raft_consensus.cc:2258] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.519312 30459 consensus_queue.cc:260] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.519624 30459 ts_tablet_manager.cc:1428] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:35:59.521085 29894 catalog_manager.cc:4841] TS 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091): tablet 292e794e507f41b3b806c7732231bf94 (table test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]) successfully deleted
I20260810 14:35:59.521528 30486 ts_tablet_manager.cc:1905] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:59.523969 30486 ts_tablet_manager.cc:1918] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:59.524092 30486 log.cc:1196] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/292e794e507f41b3b806c7732231bf94
I20260810 14:35:59.524101 30460 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: No bootstrap required, opened a new log
I20260810 14:35:59.524194 30460 ts_tablet_manager.cc:1397] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 14:35:59.524487 30486 ts_tablet_manager.cc:1939] T 292e794e507f41b3b806c7732231bf94 P e64e413128bc45748e8f8e507eb48d58: Deleting consensus metadata
I20260810 14:35:59.524703 30460 raft_consensus.cc:348] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.524830 30460 raft_consensus.cc:374] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.524865 30460 raft_consensus.cc:729] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.525005 30460 consensus_queue.cc:260] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.525272 30460 ts_tablet_manager.cc:1428] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 14:35:59.526113 29894 catalog_manager.cc:4841] TS e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699): tablet 292e794e507f41b3b806c7732231bf94 (table test-table [id=d35ed7aef3bf484a9a665a0a76b1bb55]) successfully deleted
I20260810 14:35:59.526567 30461 tablet_bootstrap.cc:492] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: No bootstrap required, opened a new log
I20260810 14:35:59.526674 30461 ts_tablet_manager.cc:1397] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20260810 14:35:59.527180 30461 raft_consensus.cc:348] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.527298 30461 raft_consensus.cc:374] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.527343 30461 raft_consensus.cc:729] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.527483 30461 consensus_queue.cc:260] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.527781 30461 ts_tablet_manager.cc:1428] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260810 14:35:59.582688 30429 tablet.cc:2369] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:59.606034 30274 tablet.cc:2369] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:59.636656 30126 tablet.cc:2369] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:59.693840 30466 raft_consensus.cc:482] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:59.694065 30466 raft_consensus.cc:504] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.695389 30466 leader_election.cc:288] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:35:59.699553 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.695673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.21.43.67:51249)
0810 14:35:59.695841 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.695860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.695982 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.696325 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.696328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.696354 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.696488 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.696495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.697439 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.697443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.698319 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.698331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.699048 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.699052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.699205 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.699229 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.699312 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.699386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 14:35:59.699585 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.695803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:51249 (local address 127.21.43.66:46699)
0810 14:35:59.695943 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.695947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.695964 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:35:59.696120 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.696123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.696179 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.696268 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.696610 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.697320 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.698458 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.698920 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.699034 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.699079 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.699347 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.699445 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:59.699681 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.695785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45279 (local address 127.21.43.65:39727)
0810 14:35:59.695915 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.695920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.696334 (+ 414us) server_negotiation.cc:407] Connection header received
0810 14:35:59.696435 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.696438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.696494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.696577 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.696911 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.697504 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.698603 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.699041 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.699194 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.699262 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.699476 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.699554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:35:59.699744 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.695673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39727 (local address 127.21.43.67:45279)
0810 14:35:59.696211 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.696227 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.696341 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.696606 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.696609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.696626 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.696755 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.696761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.697634 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.697636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.698458 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.698469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.699219 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.699223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.699336 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.699363 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.699442 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.699519 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":194,"thread_start_us":106,"threads_started":1}
I20260810 14:35:59.700253 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9291897940feb96cd7422b09e5c5" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58" is_pre_election: true
I20260810 14:35:59.700320 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9291897940feb96cd7422b09e5c5" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b" is_pre_election: true
I20260810 14:35:59.700455 30224 raft_consensus.cc:2454] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 0.
I20260810 14:35:59.700600 30076 raft_consensus.cc:2454] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 0.
I20260810 14:35:59.700884 30302 leader_election.cc:302] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, e64e413128bc45748e8f8e507eb48d58; no voters:
I20260810 14:35:59.701164 30466 raft_consensus.cc:2781] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:59.701229 30466 raft_consensus.cc:482] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:59.701277 30466 raft_consensus.cc:3037] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.702363 30466 raft_consensus.cc:504] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.702740 30466 leader_election.cc:288] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 election: Requested vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:35:59.703195 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9291897940feb96cd7422b09e5c5" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b"
I20260810 14:35:59.703205 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83a9291897940feb96cd7422b09e5c5" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58"
I20260810 14:35:59.703349 30076 raft_consensus.cc:3037] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.703372 30224 raft_consensus.cc:3037] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.704658 30224 raft_consensus.cc:2454] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 1.
I20260810 14:35:59.704668 30076 raft_consensus.cc:2454] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 1.
I20260810 14:35:59.704984 30302 leader_election.cc:302] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, e64e413128bc45748e8f8e507eb48d58, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:35:59.705200 30466 raft_consensus.cc:2781] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:59.705440 30466 raft_consensus.cc:686] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 1 LEADER]: Becoming Leader. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Running, Role: LEADER
I20260810 14:35:59.705734 30466 consensus_queue.cc:237] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.708261 29909 catalog_manager.cc:5495] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f43453585554dcba1cf4d40492d231c (127.21.43.67). New cstate: current_term: 1 leader_uuid: "3f43453585554dcba1cf4d40492d231c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } health_report { overall_health: HEALTHY } } }
Aug 10 14:35:59 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372558, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.126@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:35:59.782068 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.773162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38427 (local address 127.0.0.1:57978)
0810 14:35:59.773526 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.773540 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.773649 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.773889 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.773891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.774315 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:59.774411 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.774417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.775208 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.775211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.775934 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.775944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.776426 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.776429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.776489 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.777103 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:59.777115 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:59.778611 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:59.780422 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.780433 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.780439 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.780698 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.781047 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:59.781049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:59.781051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:59.781137 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:59.781541 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:59.781547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:59.781659 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.781763 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.781904 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":177,"threads_started":1}
I20260810 14:35:59.782140 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.773239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.21.43.126:38427)
0810 14:35:59.773419 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.773424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.773625 (+ 201us) server_negotiation.cc:407] Connection header received
0810 14:35:59.773732 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.773734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.773778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.773850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:59.774525 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.775052 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.776049 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.776290 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.776418 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.778766 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:59.778769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:59.778771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:59.778792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:59.780297 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.780865 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.780869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.780871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.780967 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:59.781251 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:59.781253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:59.781255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:59.781444 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:59.781530 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.781826 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.782000 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":85}
I20260810 14:35:59.786269 29909 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:35:59.786993 29909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:59.792817 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 3b86f36e439146fb83c7b90ef375cdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bcb37126cd94251a300c0c162a0dd9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:59.792897 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 3b86f36e439146fb83c7b90ef375cdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bcb37126cd94251a300c0c162a0dd9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:59.792884 30200 tablet_service.cc:1467] Processing CreateTablet for tablet 3b86f36e439146fb83c7b90ef375cdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bcb37126cd94251a300c0c162a0dd9f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:35:59.793306 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b86f36e439146fb83c7b90ef375cdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.793308 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b86f36e439146fb83c7b90ef375cdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.793308 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b86f36e439146fb83c7b90ef375cdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:59.796200 30460 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: Bootstrap starting.
I20260810 14:35:59.796643 30459 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: Bootstrap starting.
I20260810 14:35:59.796792 30461 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: Bootstrap starting.
I20260810 14:35:59.797219 30460 tablet_bootstrap.cc:654] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.797621 30459 tablet_bootstrap.cc:654] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.797766 30461 tablet_bootstrap.cc:654] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:59.798543 30460 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: No bootstrap required, opened a new log
I20260810 14:35:59.798648 30460 ts_tablet_manager.cc:1397] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:59.798820 30459 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: No bootstrap required, opened a new log
I20260810 14:35:59.798893 30459 ts_tablet_manager.cc:1397] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:35:59.799124 30460 raft_consensus.cc:348] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.799294 30460 raft_consensus.cc:374] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.799342 30460 raft_consensus.cc:729] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.799484 30460 consensus_queue.cc:260] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.799710 30459 raft_consensus.cc:348] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.799809 30460 ts_tablet_manager.cc:1428] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:35:59.799846 30459 raft_consensus.cc:374] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.799885 30459 raft_consensus.cc:729] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.799816 30480 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.800030 30480 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.800012 30459 consensus_queue.cc:260] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.800352 30459 ts_tablet_manager.cc:1428] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 14:35:59.800774 30461 tablet_bootstrap.cc:492] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: No bootstrap required, opened a new log
I20260810 14:35:59.800760 30466 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.800889 30466 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.800891 30461 ts_tablet_manager.cc:1397] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 14:35:59.801362 30461 raft_consensus.cc:348] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.801470 30461 raft_consensus.cc:374] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:59.801504 30461 raft_consensus.cc:729] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Initialized, Role: FOLLOWER
I20260810 14:35:59.801651 30461 consensus_queue.cc:260] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.801939 30461 ts_tablet_manager.cc:1428] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:35:59.801937 30467 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.802129 30467 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.848091 30467 raft_consensus.cc:482] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:59.848258 30467 raft_consensus.cc:504] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.849463 30467 leader_election.cc:288] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091)
I20260810 14:35:59.854235 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.850173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:43657 (local address 127.21.43.67:43091)
0810 14:35:59.850327 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.850334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.850350 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:59.850412 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.850415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.850484 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.850570 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.851163 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.851907 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853111 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.853532 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853675 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.853762 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.854031 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.854104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:59.854228 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.849848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:58741 (local address 127.21.43.65:39727)
0810 14:35:59.850090 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.850094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.850482 (+ 388us) server_negotiation.cc:407] Connection header received
0810 14:35:59.850583 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.850586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.850642 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.850718 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:59.851033 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.851898 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853126 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.853535 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.853740 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.854003 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.854095 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260810 14:35:59.854276 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.849737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:43091 (local address 127.21.43.66:43657)
0810 14:35:59.850129 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.850144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.850283 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.850677 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.850680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.850702 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.850944 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.850952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.852011 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.852014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.852980 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.852990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.853668 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.853859 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.853912 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.854001 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.854078 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":92,"thread_start_us":125,"threads_started":1}
I20260810 14:35:59.854688 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.849747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39727 (local address 127.21.43.66:58741)
0810 14:35:59.850374 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.850388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.850493 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.850732 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.850735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.850752 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:59.850902 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.850908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.852065 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.852068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.853001 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.853011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.853667 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.853671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.853821 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.853853 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.854448 (+ 595us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.854549 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":48,"thread_start_us":103,"threads_started":1}
I20260810 14:35:59.854888 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b86f36e439146fb83c7b90ef375cdec" candidate_uuid: "e64e413128bc45748e8f8e507eb48d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c" is_pre_election: true
I20260810 14:35:59.855086 30377 raft_consensus.cc:2454] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64e413128bc45748e8f8e507eb48d58 in term 0.
I20260810 14:35:59.855213 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b86f36e439146fb83c7b90ef375cdec" candidate_uuid: "e64e413128bc45748e8f8e507eb48d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b" is_pre_election: true
I20260810 14:35:59.855407 30076 raft_consensus.cc:2454] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64e413128bc45748e8f8e507eb48d58 in term 0.
I20260810 14:35:59.855544 30154 leader_election.cc:302] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, e64e413128bc45748e8f8e507eb48d58; no voters:
I20260810 14:35:59.855839 30467 raft_consensus.cc:2781] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:59.855896 30467 raft_consensus.cc:482] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:59.855924 30467 raft_consensus.cc:3037] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.856971 30467 raft_consensus.cc:504] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.857343 30467 leader_election.cc:288] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [CANDIDATE]: Term 1 election: Requested vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091)
I20260810 14:35:59.857743 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b86f36e439146fb83c7b90ef375cdec" candidate_uuid: "e64e413128bc45748e8f8e507eb48d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b"
I20260810 14:35:59.857846 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b86f36e439146fb83c7b90ef375cdec" candidate_uuid: "e64e413128bc45748e8f8e507eb48d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c"
I20260810 14:35:59.857882 30076 raft_consensus.cc:3037] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.857972 30377 raft_consensus.cc:3037] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:59.859081 30076 raft_consensus.cc:2454] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64e413128bc45748e8f8e507eb48d58 in term 1.
I20260810 14:35:59.859081 30377 raft_consensus.cc:2454] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64e413128bc45748e8f8e507eb48d58 in term 1.
I20260810 14:35:59.859428 30154 leader_election.cc:302] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, e64e413128bc45748e8f8e507eb48d58; no voters:
I20260810 14:35:59.859637 30467 raft_consensus.cc:2781] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:59.859854 30467 raft_consensus.cc:686] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [term 1 LEADER]: Becoming Leader. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Running, Role: LEADER
I20260810 14:35:59.860131 30467 consensus_queue.cc:237] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:35:59.861438 30506 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: TxnStatusTablet state changed. Reason: New leader e64e413128bc45748e8f8e507eb48d58. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.861563 30506 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:59.861860 30508 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:35:59.862071 30508 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:35:59.863123 29907 catalog_manager.cc:5495] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to e64e413128bc45748e8f8e507eb48d58 (127.21.43.66). New cstate: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } health_report { overall_health: UNKNOWN } } }
I20260810 14:35:59.893592 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.885597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.21.43.66:46699)
0810 14:35:59.885726 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.885731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.888797 (+ 3066us) server_negotiation.cc:407] Connection header received
0810 14:35:59.888958 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.888961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.889047 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.889163 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:59.889472 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.890306 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.891743 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.892143 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.892262 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.892583 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.893130 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.893245 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.893345 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.893416 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:35:59.893592 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.885480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.0.0.1:57508)
0810 14:35:59.888681 (+ 3201us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.888696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.888809 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.889204 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.889207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.889216 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:59.889341 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.889347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.890444 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.890448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.891599 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.891612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.892255 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.892259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.892351 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.892416 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.893238 (+ 822us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.893256 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.893321 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.893415 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3085}
I20260810 14:35:59.897265 30377 raft_consensus.cc:1261] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Refusing update from remote peer e64e413128bc45748e8f8e507eb48d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.897382 30076 raft_consensus.cc:1261] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Refusing update from remote peer e64e413128bc45748e8f8e507eb48d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.898032 30467 consensus_queue.cc:1035] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.898419 30467 consensus_queue.cc:1035] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.900655 30480 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: TxnStatusTablet state changed. Reason: New leader e64e413128bc45748e8f8e507eb48d58. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.900784 30480 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.902071 30466 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: TxnStatusTablet state changed. Reason: New leader e64e413128bc45748e8f8e507eb48d58. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.902197 30466 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.904119 30467 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.904237 30467 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:59.904484 30466 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.904598 30466 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P 3f43453585554dcba1cf4d40492d231c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.908865 30506 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.909029 30506 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P e64e413128bc45748e8f8e507eb48d58: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:35:59.909698 30480 tablet_replica.cc:439] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e64e413128bc45748e8f8e507eb48d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } } }
I20260810 14:35:59.909824 30480 tablet_replica.cc:442] T 3b86f36e439146fb83c7b90ef375cdec P fca47d519173443d972c7783d92aaf9b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:35:59.914830 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.909390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.21.43.67:43091)
0810 14:35:59.909508 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.909512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.909969 (+ 457us) server_negotiation.cc:407] Connection header received
0810 14:35:59.910081 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.910084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.910172 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.910277 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:59.911107 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.911979 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.913026 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.913378 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.913488 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.913747 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.914411 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.914528 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.914606 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.914670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:35:59.927027 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.923009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:53533 (local address 127.21.43.66:46699)
0810 14:35:59.923207 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.923214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.923474 (+ 260us) server_negotiation.cc:407] Connection header received
0810 14:35:59.923626 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.923628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.923677 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.923743 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:59.924070 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.924630 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.925919 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.926147 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.926236 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.926594 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.926676 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.926734 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.926852 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.926903 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:35:59.927104 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.922903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.21.43.67:53533)
0810 14:35:59.923385 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.923400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.923520 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.923779 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.923782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.923790 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:59.923922 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.923929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.924746 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.924750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.925796 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.925810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.926271 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.926275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.926374 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.926442 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.926757 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.926764 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.926833 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.926923 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":138,"threads_started":1}
I20260810 14:35:59.934754 30224 raft_consensus.cc:1261] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Refusing update from remote peer 3f43453585554dcba1cf4d40492d231c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.934794 30076 raft_consensus.cc:1261] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Refusing update from remote peer 3f43453585554dcba1cf4d40492d231c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:59.935345 30466 consensus_queue.cc:1035] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.935600 30493 consensus_queue.cc:1035] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:59.961140 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.956809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:45367 (local address 127.21.43.67:43091)
0810 14:35:59.956944 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:59.956949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:59.957270 (+ 321us) server_negotiation.cc:407] Connection header received
0810 14:35:59.957443 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:59.957447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:59.957499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:59.957546 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:59.957944 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.958530 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.959882 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.960098 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.960184 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.960507 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.960580 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.960636 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:59.960916 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 14:35:59.960982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:59.961596 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.956642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:43091 (local address 127.21.43.66:45367)
0810 14:35:59.957187 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.957203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:59.957315 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:59.957685 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:59.957688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:59.957695 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:35:59.957813 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.957819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.958656 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.958659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.959762 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:59.959774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:59.960214 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:59.960217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:59.960318 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:59.960374 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:35:59.960666 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:35:59.960673 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:35:59.961357 (+ 684us) client_negotiation.cc:240] Negotiation successful
0810 14:35:59.961440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260810 14:35:59.991125 29909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:59.991391 29909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:59.994133 29909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Sending DeleteTablet for 3 replicas of tablet f83a9291897940feb96cd7422b09e5c5
I20260810 14:35:59.994858 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet f83a9291897940feb96cd7422b09e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41926
I20260810 14:35:59.995115 30485 tablet_replica.cc:330] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: stopping tablet replica
I20260810 14:35:59.995286 30485 raft_consensus.cc:2229] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:59.995285 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet f83a9291897940feb96cd7422b09e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:58438
I20260810 14:35:59.995435 30485 raft_consensus.cc:2258] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.995504 30487 tablet_replica.cc:330] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: stopping tablet replica
I20260810 14:35:59.995512 30200 tablet_service.cc:1514] Processing DeleteTablet for tablet f83a9291897940feb96cd7422b09e5c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:59 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57498
I20260810 14:35:59.995620 30487 raft_consensus.cc:2229] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:59.995709 30486 tablet_replica.cc:330] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: stopping tablet replica
I20260810 14:35:59.995810 30486 raft_consensus.cc:2229] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:59.995915 30486 raft_consensus.cc:2258] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.995945 30487 raft_consensus.cc:2258] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:59.996874 30486 ts_tablet_manager.cc:1905] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:59.998562 30485 ts_tablet_manager.cc:1905] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:59.999671 30487 ts_tablet_manager.cc:1905] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.000145 30486 ts_tablet_manager.cc:1918] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:36:00.000221 30486 log.cc:1196] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/f83a9291897940feb96cd7422b09e5c5
I20260810 14:36:00.000479 30486 ts_tablet_manager.cc:1939] T f83a9291897940feb96cd7422b09e5c5 P e64e413128bc45748e8f8e507eb48d58: Deleting consensus metadata
I20260810 14:36:00.001296 29894 catalog_manager.cc:4841] TS e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699): tablet f83a9291897940feb96cd7422b09e5c5 (table test-table [id=49e565d8f1264d69ba6544ef58cddb17]) successfully deleted
I20260810 14:36:00.002753 30487 ts_tablet_manager.cc:1918] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:36:00.002836 30487 log.cc:1196] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/f83a9291897940feb96cd7422b09e5c5
I20260810 14:36:00.003173 30487 ts_tablet_manager.cc:1939] T f83a9291897940feb96cd7422b09e5c5 P 3f43453585554dcba1cf4d40492d231c: Deleting consensus metadata
I20260810 14:36:00.003998 29894 catalog_manager.cc:4841] TS 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091): tablet f83a9291897940feb96cd7422b09e5c5 (table test-table [id=49e565d8f1264d69ba6544ef58cddb17]) successfully deleted
I20260810 14:36:00.006047 30485 ts_tablet_manager.cc:1918] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:36:00.006127 30485 log.cc:1196] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/f83a9291897940feb96cd7422b09e5c5
I20260810 14:36:00.006410 30485 ts_tablet_manager.cc:1939] T f83a9291897940feb96cd7422b09e5c5 P fca47d519173443d972c7783d92aaf9b: Deleting consensus metadata
I20260810 14:36:00.007285 29895 catalog_manager.cc:4841] TS fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727): tablet f83a9291897940feb96cd7422b09e5c5 (table test-table [id=49e565d8f1264d69ba6544ef58cddb17]) successfully deleted
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[29862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:36:00.031548 29962 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:00.024631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.21.43.126:38427)
0810 14:36:00.024778 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.024782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.025045 (+ 263us) server_negotiation.cc:407] Connection header received
0810 14:36:00.025213 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.025219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.025276 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.025369 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:00.026122 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.026807 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.027768 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.027992 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.028104 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.031391 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57984: BlockingRecv error: recv got EOF from 127.0.0.1:57984 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[29862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM
I20260810 14:36:00.043704 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.033159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.21.43.126:38427)
0810 14:36:00.033289 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.033293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.034678 (+ 1385us) server_negotiation.cc:407] Connection header received
0810 14:36:00.034836 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.034842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.034891 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.034978 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:00.035763 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.036356 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.037199 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.037493 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.037606 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.040317 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:00.040322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:00.040330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:00.040357 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:00.041890 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:00.042362 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:00.042366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:00.042367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:00.042446 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:00.042737 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:00.042742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:00.042746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:00.042953 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:00.043096 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:00.043379 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:36:00.043503 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:36:00.046702 29909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:00.047250 29909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:00.052386 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (DEFAULT_TABLE table=test-table [id=9807efeffa5e4e849efa6bb147a242a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.052762 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.052850 30200 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (DEFAULT_TABLE table=test-table [id=9807efeffa5e4e849efa6bb147a242a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.053145 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.055831 30461 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Bootstrap starting.
I20260810 14:36:00.056309 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (DEFAULT_TABLE table=test-table [id=9807efeffa5e4e849efa6bb147a242a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.056420 30460 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Bootstrap starting.
I20260810 14:36:00.056599 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.057365 30460 tablet_bootstrap.cc:654] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.057830 30461 tablet_bootstrap.cc:654] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.059887 30461 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: No bootstrap required, opened a new log
I20260810 14:36:00.059983 30461 ts_tablet_manager.cc:1397] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 14:36:00.060484 30461 raft_consensus.cc:348] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.060663 30461 raft_consensus.cc:374] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.060751 30461 raft_consensus.cc:729] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.060905 30461 consensus_queue.cc:260] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.062140 30461 ts_tablet_manager.cc:1428] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:36:00.063644 30459 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Bootstrap starting.
I20260810 14:36:00.064703 30460 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: No bootstrap required, opened a new log
I20260810 14:36:00.064756 30459 tablet_bootstrap.cc:654] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.064800 30460 ts_tablet_manager.cc:1397] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260810 14:36:00.065213 30460 raft_consensus.cc:348] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.065326 30460 raft_consensus.cc:374] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.065356 30460 raft_consensus.cc:729] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.065485 30460 consensus_queue.cc:260] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.065755 30460 ts_tablet_manager.cc:1428] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:36:00.066483 30459 tablet_bootstrap.cc:492] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: No bootstrap required, opened a new log
I20260810 14:36:00.066589 30459 ts_tablet_manager.cc:1397] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:36:00.067034 30459 raft_consensus.cc:348] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.067190 30459 raft_consensus.cc:374] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.067224 30459 raft_consensus.cc:729] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.067354 30459 consensus_queue.cc:260] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.067627 30459 ts_tablet_manager.cc:1428] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:36:00.168526 30523 raft_consensus.cc:482] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:00.168674 30523 raft_consensus.cc:504] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.169210 30523 leader_election.cc:288] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:36:00.169739 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ebea41f6c4ccaa5c5d224e4f2f432" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b" is_pre_election: true
I20260810 14:36:00.169745 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ebea41f6c4ccaa5c5d224e4f2f432" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58" is_pre_election: true
I20260810 14:36:00.169914 30224 raft_consensus.cc:2454] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 0.
I20260810 14:36:00.169914 30076 raft_consensus.cc:2454] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 0.
I20260810 14:36:00.170236 30303 leader_election.cc:302] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:36:00.170423 30523 raft_consensus.cc:2781] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:00.170475 30523 raft_consensus.cc:482] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:00.170507 30523 raft_consensus.cc:3037] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.171793 30523 raft_consensus.cc:504] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.172173 30523 leader_election.cc:288] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 election: Requested vote from peers fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727), e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699)
I20260810 14:36:00.172557 30076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ebea41f6c4ccaa5c5d224e4f2f432" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca47d519173443d972c7783d92aaf9b"
I20260810 14:36:00.172580 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ebea41f6c4ccaa5c5d224e4f2f432" candidate_uuid: "3f43453585554dcba1cf4d40492d231c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58"
I20260810 14:36:00.172691 30076 raft_consensus.cc:3037] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.172715 30224 raft_consensus.cc:3037] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.173612 30224 raft_consensus.cc:2454] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 1.
I20260810 14:36:00.173709 30076 raft_consensus.cc:2454] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f43453585554dcba1cf4d40492d231c in term 1.
I20260810 14:36:00.173976 30302 leader_election.cc:302] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f43453585554dcba1cf4d40492d231c, e64e413128bc45748e8f8e507eb48d58; no voters:
I20260810 14:36:00.174191 30523 raft_consensus.cc:2781] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:00.174292 30523 raft_consensus.cc:686] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 1 LEADER]: Becoming Leader. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Running, Role: LEADER
I20260810 14:36:00.174443 30523 consensus_queue.cc:237] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.176121 29909 catalog_manager.cc:5495] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f43453585554dcba1cf4d40492d231c (127.21.43.67). New cstate: current_term: 1 leader_uuid: "3f43453585554dcba1cf4d40492d231c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } health_report { overall_health: HEALTHY } } }
I20260810 14:36:00.185688 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.182512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.21.43.67:43091)
0810 14:36:00.182653 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.182659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.182674 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:00.182827 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.182830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.182882 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.182966 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:00.183242 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.183750 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.184610 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.184828 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.184939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.185210 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.185280 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.185361 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:00.185477 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:36:00.185539 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:36:00.187705 30224 raft_consensus.cc:1261] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Refusing update from remote peer 3f43453585554dcba1cf4d40492d231c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:00.187716 30076 raft_consensus.cc:1261] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Refusing update from remote peer 3f43453585554dcba1cf4d40492d231c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:00.188135 30493 consensus_queue.cc:1035] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:00.188310 30523 consensus_queue.cc:1035] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:00.196548 29909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:00.196704 29909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:00.198599 29909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Sending DeleteTablet for 3 replicas of tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432
I20260810 14:36:00.199186 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41926
I20260810 14:36:00.199198 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:58438
I20260810 14:36:00.199373 30485 tablet_replica.cc:330] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: stopping tablet replica
I20260810 14:36:00.199411 30200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57498
I20260810 14:36:00.199510 30485 raft_consensus.cc:2229] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:00.199596 30487 tablet_replica.cc:330] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: stopping tablet replica
I20260810 14:36:00.199627 30486 tablet_replica.cc:330] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: stopping tablet replica
I20260810 14:36:00.199676 30485 raft_consensus.cc:2258] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.199731 30487 raft_consensus.cc:2229] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:00.199753 30486 raft_consensus.cc:2229] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:00.199874 30486 raft_consensus.cc:2258] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.199921 30487 raft_consensus.cc:2258] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.200543 30487 ts_tablet_manager.cc:1905] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.200635 30486 ts_tablet_manager.cc:1905] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.200573 29909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:00.200986 29909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:00.202158 30485 ts_tablet_manager.cc:1905] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.203116 30487 ts_tablet_manager.cc:1918] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:00.203184 30486 ts_tablet_manager.cc:1918] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:00.203262 30486 log.cc:1196] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/8c7ebea41f6c4ccaa5c5d224e4f2f432
I20260810 14:36:00.203267 30487 log.cc:1196] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/8c7ebea41f6c4ccaa5c5d224e4f2f432
I20260810 14:36:00.203541 30486 ts_tablet_manager.cc:1939] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P e64e413128bc45748e8f8e507eb48d58: Deleting consensus metadata
I20260810 14:36:00.203617 30487 ts_tablet_manager.cc:1939] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P 3f43453585554dcba1cf4d40492d231c: Deleting consensus metadata
I20260810 14:36:00.204222 29894 catalog_manager.cc:4841] TS e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699): tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (table test-table [id=9807efeffa5e4e849efa6bb147a242a7]) successfully deleted
I20260810 14:36:00.204380 29894 catalog_manager.cc:4841] TS 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091): tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (table test-table [id=9807efeffa5e4e849efa6bb147a242a7]) successfully deleted
I20260810 14:36:00.204591 30485 ts_tablet_manager.cc:1918] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:00.204643 30485 log.cc:1196] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/8c7ebea41f6c4ccaa5c5d224e4f2f432
I20260810 14:36:00.204824 30485 ts_tablet_manager.cc:1939] T 8c7ebea41f6c4ccaa5c5d224e4f2f432 P fca47d519173443d972c7783d92aaf9b: Deleting consensus metadata
I20260810 14:36:00.205526 29895 catalog_manager.cc:4841] TS fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727): tablet 8c7ebea41f6c4ccaa5c5d224e4f2f432 (table test-table [id=9807efeffa5e4e849efa6bb147a242a7]) successfully deleted
I20260810 14:36:00.206139 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 (DEFAULT_TABLE table=test-table [id=0351e18272b747e0bd8d368f229c9b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.206297 30200 tablet_service.cc:1467] Processing CreateTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 (DEFAULT_TABLE table=test-table [id=0351e18272b747e0bd8d368f229c9b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.206406 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ddef3099d4012b161ddbfda91a0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.206575 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ddef3099d4012b161ddbfda91a0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.207417 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 (DEFAULT_TABLE table=test-table [id=0351e18272b747e0bd8d368f229c9b76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:00.207675 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159ddef3099d4012b161ddbfda91a0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:00.208515 30460 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: Bootstrap starting.
I20260810 14:36:00.208674 30461 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Bootstrap starting.
I20260810 14:36:00.209409 30460 tablet_bootstrap.cc:654] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.209473 30461 tablet_bootstrap.cc:654] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.209872 30459 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Bootstrap starting.
I20260810 14:36:00.210764 30459 tablet_bootstrap.cc:654] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:00.210866 30460 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: No bootstrap required, opened a new log
I20260810 14:36:00.210971 30460 ts_tablet_manager.cc:1397] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 14:36:00.211048 30461 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: No bootstrap required, opened a new log
I20260810 14:36:00.211127 30461 ts_tablet_manager.cc:1397] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:36:00.211421 30460 raft_consensus.cc:348] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.211552 30460 raft_consensus.cc:374] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.211520 30461 raft_consensus.cc:348] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.211592 30460 raft_consensus.cc:729] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.211623 30461 raft_consensus.cc:374] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.211658 30461 raft_consensus.cc:729] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64e413128bc45748e8f8e507eb48d58, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.211741 30460 consensus_queue.cc:260] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.211786 30461 consensus_queue.cc:260] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.211911 30459 tablet_bootstrap.cc:492] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: No bootstrap required, opened a new log
I20260810 14:36:00.211999 30459 ts_tablet_manager.cc:1397] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:00.212046 30461 ts_tablet_manager.cc:1428] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:36:00.212046 30460 ts_tablet_manager.cc:1428] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:36:00.212387 30459 raft_consensus.cc:348] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.212504 30459 raft_consensus.cc:374] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:00.212550 30459 raft_consensus.cc:729] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f43453585554dcba1cf4d40492d231c, State: Initialized, Role: FOLLOWER
I20260810 14:36:00.212702 30459 consensus_queue.cc:260] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.212998 30459 ts_tablet_manager.cc:1428] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:36:00.261417 30480 raft_consensus.cc:482] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:00.261572 30480 raft_consensus.cc:504] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.262035 30480 leader_election.cc:288] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699), 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091)
I20260810 14:36:00.262499 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ddef3099d4012b161ddbfda91a0d0" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58" is_pre_election: true
I20260810 14:36:00.262641 30224 raft_consensus.cc:2454] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 0.
I20260810 14:36:00.262617 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ddef3099d4012b161ddbfda91a0d0" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c" is_pre_election: true
I20260810 14:36:00.262812 30377 raft_consensus.cc:2454] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 0.
I20260810 14:36:00.262991 30006 leader_election.cc:302] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64e413128bc45748e8f8e507eb48d58, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:36:00.263198 30480 raft_consensus.cc:2781] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:00.263264 30480 raft_consensus.cc:482] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:00.263307 30480 raft_consensus.cc:3037] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.264309 30480 raft_consensus.cc:504] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.264629 30480 leader_election.cc:288] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 election: Requested vote from peers e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699), 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091)
I20260810 14:36:00.264977 30224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ddef3099d4012b161ddbfda91a0d0" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64e413128bc45748e8f8e507eb48d58"
I20260810 14:36:00.265048 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159ddef3099d4012b161ddbfda91a0d0" candidate_uuid: "fca47d519173443d972c7783d92aaf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f43453585554dcba1cf4d40492d231c"
I20260810 14:36:00.265122 30224 raft_consensus.cc:3037] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.265174 30377 raft_consensus.cc:3037] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:00.266039 30224 raft_consensus.cc:2454] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 1.
I20260810 14:36:00.266161 30377 raft_consensus.cc:2454] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca47d519173443d972c7783d92aaf9b in term 1.
I20260810 14:36:00.266324 30006 leader_election.cc:302] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e64e413128bc45748e8f8e507eb48d58, fca47d519173443d972c7783d92aaf9b; no voters:
I20260810 14:36:00.266536 30480 raft_consensus.cc:2781] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:00.266772 30480 raft_consensus.cc:686] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 1 LEADER]: Becoming Leader. State: Replica: fca47d519173443d972c7783d92aaf9b, State: Running, Role: LEADER
I20260810 14:36:00.266948 30480 consensus_queue.cc:237] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } }
I20260810 14:36:00.268692 29908 catalog_manager.cc:5495] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b reported cstate change: term changed from 0 to 1, leader changed from <none> to fca47d519173443d972c7783d92aaf9b (127.21.43.65). New cstate: current_term: 1 leader_uuid: "fca47d519173443d972c7783d92aaf9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca47d519173443d972c7783d92aaf9b" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 } health_report { overall_health: UNKNOWN } } }
I20260810 14:36:00.286180 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.282991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.21.43.65:39727)
0810 14:36:00.283119 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.283125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.283168 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:36:00.283363 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.283366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.283421 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.283503 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:00.283762 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.284275 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.285114 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.285354 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.285470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.285735 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.285814 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.285896 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:00.285990 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 14:36:00.286046 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:36:00.296811 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.293628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:53185 (local address 127.21.43.66:46699)
0810 14:36:00.293757 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.293760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.293914 (+ 154us) server_negotiation.cc:407] Connection header received
0810 14:36:00.294067 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.294072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.294133 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.294209 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:00.294467 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.295045 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.295918 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.296144 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.296224 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.296468 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.296534 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.296586 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:00.296664 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:36:00.296705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:36:00.296829 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.293491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46699 (local address 127.21.43.65:53185)
0810 14:36:00.293836 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.293849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:00.293938 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:00.294262 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:00.294265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:00.294274 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:36:00.294372 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:00.294376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.295165 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.295167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:00.295817 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:00.295827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.296251 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.296253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:00.296312 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.296355 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.296604 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.296609 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:36:00.296657 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:36:00.296716 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
I20260810 14:36:00.303344 30377 raft_consensus.cc:1261] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Refusing update from remote peer fca47d519173443d972c7783d92aaf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:00.303395 30224 raft_consensus.cc:1261] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Refusing update from remote peer fca47d519173443d972c7783d92aaf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:00.303762 30550 consensus_queue.cc:1035] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f43453585554dcba1cf4d40492d231c" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:00.303894 30480 consensus_queue.cc:1035] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64e413128bc45748e8f8e507eb48d58" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:00.320979 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.317749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39727 (local address 127.21.43.66:35309)
0810 14:36:00.317900 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.317915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:00.318001 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:00.318275 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:00.318278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:00.318285 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:36:00.318377 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:00.318381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.319131 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.319155 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:00.319737 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:00.319745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.320215 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.320218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:00.320302 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.320338 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.320655 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.320662 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:36:00.320766 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 14:36:00.320854 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 14:36:00.320979 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.317830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:35309 (local address 127.21.43.65:39727)
0810 14:36:00.317930 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:00.317936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:00.317954 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:00.318111 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:00.318114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:00.318164 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:00.318240 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:00.318492 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.319014 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.319870 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:00.320103 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:00.320214 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:00.320459 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:00.320558 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:00.320635 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:00.320784 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 14:36:00.320837 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:36:00.341708 29908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:00.341897 29908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:00.344056 29908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414209e6550442909a9fa008d7645a67: Sending DeleteTablet for 3 replicas of tablet 159ddef3099d4012b161ddbfda91a0d0
I20260810 14:36:00.344635 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41926
I20260810 14:36:00.344708 30200 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57498
I20260810 14:36:00.344844 30485 tablet_replica.cc:330] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: stopping tablet replica
I20260810 14:36:00.344887 30486 tablet_replica.cc:330] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: stopping tablet replica
I20260810 14:36:00.344980 30485 raft_consensus.cc:2229] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:00.344990 30486 raft_consensus.cc:2229] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:00.345103 30486 raft_consensus.cc:2258] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.345188 30485 raft_consensus.cc:2258] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.346014 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29987
I20260810 14:36:00.346009 30485 ts_tablet_manager.cc:1905] T 159ddef3099d4012b161ddbfda91a0d0 P fca47d519173443d972c7783d92aaf9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.347837 30486 ts_tablet_manager.cc:1905] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.351209 30486 ts_tablet_manager.cc:1918] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:36:00.351312 30486 log.cc:1196] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/159ddef3099d4012b161ddbfda91a0d0
I20260810 14:36:00.351593 30486 ts_tablet_manager.cc:1939] T 159ddef3099d4012b161ddbfda91a0d0 P e64e413128bc45748e8f8e507eb48d58: Deleting consensus metadata
I20260810 14:36:00.351810 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 159ddef3099d4012b161ddbfda91a0d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:00 UTC) from {username='oryx', principal='oryx/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:58438
I20260810 14:36:00.352056 30487 tablet_replica.cc:330] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: stopping tablet replica
I20260810 14:36:00.352190 30487 raft_consensus.cc:2229] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:00.352329 30487 raft_consensus.cc:2258] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:00.352828 29894 catalog_manager.cc:4841] TS e64e413128bc45748e8f8e507eb48d58 (127.21.43.66:46699): tablet 159ddef3099d4012b161ddbfda91a0d0 (table test-table [id=0351e18272b747e0bd8d368f229c9b76]) successfully deleted
I20260810 14:36:00.353173 30487 ts_tablet_manager.cc:1905] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:00.355624 30487 ts_tablet_manager.cc:1918] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:36:00.355722 30487 log.cc:1196] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/159ddef3099d4012b161ddbfda91a0d0
I20260810 14:36:00.355993 30487 ts_tablet_manager.cc:1939] T 159ddef3099d4012b161ddbfda91a0d0 P 3f43453585554dcba1cf4d40492d231c: Deleting consensus metadata
I20260810 14:36:00.356693 29894 catalog_manager.cc:4841] TS 3f43453585554dcba1cf4d40492d231c (127.21.43.67:43091): tablet 159ddef3099d4012b161ddbfda91a0d0 (table test-table [id=0351e18272b747e0bd8d368f229c9b76]) successfully deleted
W20260810 14:36:00.361375 29895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:39727 (error 108)
I20260810 14:36:00.361563 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 30136
I20260810 14:36:00.363987 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.363206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39727 (local address 127.0.0.1:41950)
0810 14:36:00.363776 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.363862 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":62,"threads_started":1}
W20260810 14:36:00.364090 29895 catalog_manager.cc:4584] TS fca47d519173443d972c7783d92aaf9b (127.21.43.65:39727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 159ddef3099d4012b161ddbfda91a0d0: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39727: connect: Connection refused (error 111)
I20260810 14:36:00.372495 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 30284
I20260810 14:36:00.387106 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 29875
2026-08-10T14:36:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3246 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[30566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[30566](info): set up 2 sockets
Aug 10 14:36:00 dist-test-slave-chsk krb5kdc[30566](info): commencing operation
krb5kdc: starting...
W20260810 14:36:01.491379 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:36:01 dist-test-slave-chsk krb5kdc[30566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:36:01.621742 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:45687
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:45687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:01.744050 30580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:01.744414 30580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:01.744503 30580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:01.748838 30580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:01.748926 30580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:01.748965 30580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:01.749024 30580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:01.749085 30580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:01.754804 30580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:45687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:45687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:01.756064 30580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:01.757160 30580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:01.766094 30589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:01.766284 30580 server_base.cc:1034] running on GCE node
W20260810 14:36:01.766094 30587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:01.766119 30586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:01.767020 30580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:01.768179 30580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:01.769404 30580 hybrid_clock.cc:648] HybridClock initialized: now 1786372561769371 us; error 45 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T14:36:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1390 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:01 dist-test-slave-chsk krb5kdc[30594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:01 dist-test-slave-chsk krb5kdc[30594](info): set up 2 sockets
Aug 10 14:36:01 dist-test-slave-chsk krb5kdc[30594](info): commencing operation
krb5kdc: starting...
W20260810 14:36:02.879482 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:36:02 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:36:03.011340 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43339
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:03.124328 30608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:03.124640 30608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:03.124732 30608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:03.128829 30608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:03.128916 30608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:03.128954 30608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:03.129022 30608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:03.129060 30608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:03.135509 30608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:03.136638 30608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:03.137619 30608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:03.146143 30615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.146224 30614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.146162 30619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:03.146397 30608 server_base.cc:1034] running on GCE node
I20260810 14:36:03.146972 30608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:03.148150 30608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:03.149333 30608 hybrid_clock.cc:648] HybridClock initialized: now 1786372563149300 us; error 44 us; skew 500 ppm
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:03.152177 30608 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:36:03.152993 30608 webserver.cc:460] Webserver started at http://127.21.43.126:38317/ using document root <none> and password file <none>
I20260810 14:36:03.153501 30608 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:03.153560 30608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:03.153725 30608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:03.155754 30608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "07e05a6c89a6455a8f0b5d4b788ce486"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "acfa6f95e4a8060d2000a0544b76311f"
server_key_iv: "6118cf6d4225ed35cfd8d8ae77cc810d"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.156268 30608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "07e05a6c89a6455a8f0b5d4b788ce486"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "acfa6f95e4a8060d2000a0544b76311f"
server_key_iv: "6118cf6d4225ed35cfd8d8ae77cc810d"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.159896 30608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:03.162220 30626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.163496 30608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:03.163625 30608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "07e05a6c89a6455a8f0b5d4b788ce486"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "acfa6f95e4a8060d2000a0544b76311f"
server_key_iv: "6118cf6d4225ed35cfd8d8ae77cc810d"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.163730 30608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:03.179952 30608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:03.183015 30608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:03.183264 30608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:03.190691 30608 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43339
I20260810 14:36:03.190717 30683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43339 every 8 connection(s)
I20260810 14:36:03.191797 30608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:03.194550 30684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:03.198520 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 30608
I20260810 14:36:03.198680 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:36:03.198956 21677 external_mini_cluster.cc:1428] Setting key 86d045bfce822c270a2a8a7e615c1b35
I20260810 14:36:03.200618 30684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Bootstrap starting.
I20260810 14:36:03.202944 30684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:03.203764 30684 log.cc:824] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:03.205722 30684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: No bootstrap required, opened a new log
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:03.209702 30684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:03.210014 30684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:03.210116 30684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e05a6c89a6455a8f0b5d4b788ce486, State: Initialized, Role: FOLLOWER
I20260810 14:36:03.210618 30684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:03.210749 30684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:03.210832 30684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:03.210933 30684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:03.211911 30684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:03.212281 30684 leader_election.cc:302] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e05a6c89a6455a8f0b5d4b788ce486; no voters:
I20260810 14:36:03.212621 30684 leader_election.cc:288] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:03.212704 30689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:03.212937 30689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 LEADER]: Becoming Leader. State: Replica: 07e05a6c89a6455a8f0b5d4b788ce486, State: Running, Role: LEADER
I20260810 14:36:03.213323 30689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:03.214090 30684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:03.214548 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.200697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.21.43.126:43339)
0810 14:36:03.201176 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:03.201192 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:03.201228 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:36:03.201918 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:03.201936 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:03.202296 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:03.202627 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:03.203570 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.204657 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.205552 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.206072 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.206238 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.208905 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:03.208910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:03.208924 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:03.208952 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:03.211015 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.211504 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.211509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.211516 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.211647 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.211948 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.211952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.211953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.212456 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:03.212687 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:03.213352 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 14:36:03.213666 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":151,"threads_started":1}
I20260810 14:36:03.214578 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } } }
I20260810 14:36:03.214687 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:03.215378 30690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e05a6c89a6455a8f0b5d4b788ce486. Latest consensus state: current_term: 1 leader_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } } }
I20260810 14:36:03.215490 30690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:03.216794 30700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:03.219624 30700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:03.225726 30700 catalog_manager.cc:1347] Generated new cluster ID: db81ec08838841fda32e57b802a61f90
I20260810 14:36:03.225814 30700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:03.245530 30700 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:03.246846 30700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:03.254200 30700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Generated new TSK 0
I20260810 14:36:03.255030 30700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:36:03.294555 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:03.407850 30711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:03.408152 30711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:03.408246 30711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:03.412310 30711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:03.412395 30711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:03.412508 30711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:36:03.418035 30711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:03.419168 30711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:03.420215 30711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:36:03.429955 30711 server_base.cc:1034] running on GCE node
W20260810 14:36:03.429888 30717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.429888 30719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.429885 30716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:03.430691 30711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:03.431432 30711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:03.432624 30711 hybrid_clock.cc:648] HybridClock initialized: now 1786372563432612 us; error 26 us; skew 500 ppm
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:03.436112 30711 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:36:03.437018 30711 webserver.cc:460] Webserver started at http://127.21.43.65:45731/ using document root <none> and password file <none>
I20260810 14:36:03.437583 30711 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:03.437678 30711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:03.437908 30711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:03.440006 30711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "ebe27668b6fe4ae0942373a10767a092"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "bd9ab9e7c1f74d7ddea5d4d4ca7180a9"
server_key_iv: "98b67c7ed972cade4c2a9549f5ec7740"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.440583 30711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe27668b6fe4ae0942373a10767a092"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "bd9ab9e7c1f74d7ddea5d4d4ca7180a9"
server_key_iv: "98b67c7ed972cade4c2a9549f5ec7740"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.444491 30711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:36:03.446820 30726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.448009 30711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:03.448167 30711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "ebe27668b6fe4ae0942373a10767a092"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "bd9ab9e7c1f74d7ddea5d4d4ca7180a9"
server_key_iv: "98b67c7ed972cade4c2a9549f5ec7740"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.448285 30711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:03.475185 30711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:03.478627 30711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:03.478844 30711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:03.479506 30711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:03.480365 30711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:03.480438 30711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.480512 30711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:03.480568 30711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.489464 30711 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:36443
I20260810 14:36:03.489488 30852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:36443 every 8 connection(s)
I20260810 14:36:03.490370 30711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:03.492429 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 30711
I20260810 14:36:03.492570 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:03.492882 21677 external_mini_cluster.cc:1428] Setting key 97b093cdebdd6757f48ffefee05baa83
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:36:03.504565 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.492237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:47925 (local address 127.21.43.126:43339)
0810 14:36:03.492374 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:03.492378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:03.493708 (+ 1330us) server_negotiation.cc:407] Connection header received
0810 14:36:03.494689 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:03.494693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:03.494746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:03.494824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:03.495988 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.496521 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.497647 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.497882 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.497995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.500482 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:03.500485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:03.500487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:03.500511 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:03.502311 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.502885 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.502890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.502892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.502985 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.503397 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.503401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.503404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.503640 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:03.503761 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:03.504263 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 14:36:03.504395 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:36:03.505509 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.492728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:47925)
0810 14:36:03.493556 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.493606 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:03.494483 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:03.494989 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:03.494995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:03.495373 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:03.495838 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.495850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.496663 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.496669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.497510 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.497519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.498032 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.498036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.498207 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.498808 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:03.498824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:03.500312 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:03.502442 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.502447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.502458 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.502777 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.503085 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.503088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.503089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.503294 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.503757 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:03.503763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:03.504004 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 14:36:03.504204 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 14:36:03.504567 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:36:03.506811 30853 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43339
I20260810 14:36:03.507064 30853 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:03.507589 30853 heartbeater.cc:511] Master 127.21.43.126:43339 requested a full tablet report, sending...
I20260810 14:36:03.509052 30643 ts_manager.cc:194] Registered new tserver with Master: ebe27668b6fe4ae0942373a10767a092 (127.21.43.65:36443)
I20260810 14:36:03.510363 30643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:47925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:36:03.522475 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:03.637020 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:03.637324 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:03.637387 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:03.641724 30859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:03.641798 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:03.641857 30859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:36:03.647296 30859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:03.648327 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:03.649317 30859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:03.658627 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.658704 30869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.658705 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:03.659013 30859 server_base.cc:1034] running on GCE node
I20260810 14:36:03.659428 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:03.660051 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:03.661249 30859 hybrid_clock.cc:648] HybridClock initialized: now 1786372563661236 us; error 24 us; skew 500 ppm
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:03.664053 30859 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:36:03.664938 30859 webserver.cc:460] Webserver started at http://127.21.43.66:44077/ using document root <none> and password file <none>
I20260810 14:36:03.665498 30859 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:03.665592 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:03.665824 30859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:03.667963 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "1028e9cffca5430282e6c0236b9d5309"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "ef605f5cc212026bec0823c3dc4deb37"
server_key_iv: "4fbbcbc2d24ceb9cdb69f42110eabe3c"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.668533 30859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "1028e9cffca5430282e6c0236b9d5309"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "ef605f5cc212026bec0823c3dc4deb37"
server_key_iv: "4fbbcbc2d24ceb9cdb69f42110eabe3c"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.672119 30859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:03.674461 30876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.675712 30859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:03.675848 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "1028e9cffca5430282e6c0236b9d5309"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "ef605f5cc212026bec0823c3dc4deb37"
server_key_iv: "4fbbcbc2d24ceb9cdb69f42110eabe3c"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.675958 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:03.688544 30859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:03.691308 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:03.691511 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:03.692121 30859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:03.692941 30859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:03.693027 30859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.693099 30859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:03.693135 30859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.702484 30859 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:33315
I20260810 14:36:03.702528 31002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:33315 every 8 connection(s)
I20260810 14:36:03.703536 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:03.709887 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 30859
I20260810 14:36:03.710043 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:03.710356 21677 external_mini_cluster.cc:1428] Setting key c54a7576e8382841c62209e9f667c11d
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:03.719018 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.705516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:41771 (local address 127.21.43.126:43339)
0810 14:36:03.705673 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:03.705677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:03.706807 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 14:36:03.707685 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:03.707688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:03.707737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:03.707818 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:03.708970 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.709500 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.711074 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.711391 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.711507 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.714556 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:03.714563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:03.714567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:03.714610 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:03.716595 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.717122 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.717129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.717134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.717246 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.717641 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.717648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.717651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.717924 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:03.718066 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:03.718612 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 14:36:03.718763 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:36:03.719756 31006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.705988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:41771)
0810 14:36:03.706663 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.706703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:03.707462 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:03.707973 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:03.707979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:03.708340 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:03.708814 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.708827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.709678 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.709683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.710944 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.710953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.711547 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.711552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.711722 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.712345 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:03.712372 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:03.714327 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:03.716721 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.716725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.716738 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.717004 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.717384 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.717387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.717388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.717548 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.718091 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:03.718096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:03.718334 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 14:36:03.718503 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 14:36:03.718793 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":117,"threads_started":1}
I20260810 14:36:03.721073 31003 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43339
I20260810 14:36:03.721339 31003 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:03.721850 31003 heartbeater.cc:511] Master 127.21.43.126:43339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:36:03.722935 30643 ts_manager.cc:194] Registered new tserver with Master: 1028e9cffca5430282e6c0236b9d5309 (127.21.43.66:33315)
I20260810 14:36:03.723590 30643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:41771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:36:03.742385 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:03.855530 31009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:03.855786 31009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:03.855830 31009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:03.859908 31009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:03.859973 31009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:03.860033 31009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:36:03.865301 31009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:03.866319 31009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:03.867372 31009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:03.876945 31014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.876924 31017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:03.876927 31015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:03.877215 31009 server_base.cc:1034] running on GCE node
I20260810 14:36:03.877739 31009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:03.878404 31009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:03.879594 31009 hybrid_clock.cc:648] HybridClock initialized: now 1786372563879578 us; error 53 us; skew 500 ppm
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:03.886530 31009 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:36:03.887519 31009 webserver.cc:460] Webserver started at http://127.21.43.67:39535/ using document root <none> and password file <none>
I20260810 14:36:03.888058 31009 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:03.888127 31009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:03.888300 31009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:03.890313 31009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "153dee6e00064e469e35a9aeb8aee6f2"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "981ec2c81d2017fdab967f4417a16261"
server_key_iv: "5000dbed66964d6af4f4a0fa332716ba"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.890844 31009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "153dee6e00064e469e35a9aeb8aee6f2"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "981ec2c81d2017fdab967f4417a16261"
server_key_iv: "5000dbed66964d6af4f4a0fa332716ba"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.894640 31009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:36:03.897220 31025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.898479 31009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:03.898578 31009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "153dee6e00064e469e35a9aeb8aee6f2"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "981ec2c81d2017fdab967f4417a16261"
server_key_iv: "5000dbed66964d6af4f4a0fa332716ba"
server_key_version: "encryptionkey@0"
I20260810 14:36:03.898666 31009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:03.909938 31009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:03.912988 31009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:03.913208 31009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:03.913918 31009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:03.914920 31009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:03.914970 31009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.915012 31009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:03.915037 31009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:03.924486 31009 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:35661
I20260810 14:36:03.924515 31154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:35661 every 8 connection(s)
I20260810 14:36:03.925436 31009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:03.929448 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31009
I20260810 14:36:03.929620 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:03.929945 21677 external_mini_cluster.cc:1428] Setting key b234e8e2370a3dd781bc556e3d8b484b
Aug 10 14:36:03 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:03.939757 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.927328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:36255 (local address 127.21.43.126:43339)
0810 14:36:03.927490 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:03.927495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:03.928585 (+ 1090us) server_negotiation.cc:407] Connection header received
0810 14:36:03.929634 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:03.929639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:03.929711 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:03.929789 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:03.931412 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.932187 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.933366 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.933612 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.933708 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.936246 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:03.936249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:03.936251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:03.936274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:03.937787 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.938284 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.938286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.938287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.938364 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:03.938657 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:03.938659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:03.938660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:03.938833 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:03.938906 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:03.939518 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 14:36:03.939631 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:36:03.940577 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.927742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:36255)
0810 14:36:03.928428 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.928469 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:03.929360 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:03.930003 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:03.930014 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:03.930502 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:03.931251 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.931272 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.932342 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.932344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.933214 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:03.933224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:03.933754 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:03.933756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:03.933895 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:03.934448 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:03.934466 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:03.936089 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:03.937900 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.937904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.937916 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.938188 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.938439 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:03.938441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:03.938442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:03.938572 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:03.938945 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:03.938951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:03.939290 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 14:36:03.939451 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 14:36:03.939719 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":117,"threads_started":1}
I20260810 14:36:03.941802 31155 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43339
I20260810 14:36:03.942065 31155 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:03.942633 31155 heartbeater.cc:511] Master 127.21.43.126:43339 requested a full tablet report, sending...
I20260810 14:36:03.943791 30643 ts_manager.cc:194] Registered new tserver with Master: 153dee6e00064e469e35a9aeb8aee6f2 (127.21.43.67:35661)
I20260810 14:36:03.944425 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:03.944610 30643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:36255
WARNING: no policy specified for oryx/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
Entry for principal oryx/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab.
WARNING: no policy specified for oryx/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
Entry for principal oryx/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab.
WARNING: no policy specified for oryx/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
Entry for principal oryx/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab.
I20260810 14:36:04.025895 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 30711
I20260810 14:36:04.037876 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 30859
I20260810 14:36:04.046103 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31009
I20260810 14:36:04.054316 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 30608
I20260810 14:36:04.063026 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43339
--webserver_interface=127.21.43.126
--webserver_port=38317
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:04.193100 31165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:04.193420 31165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:04.193464 31165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:04.197685 31165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:04.197762 31165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:04.197781 31165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:04.197798 31165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:04.197813 31165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:04.203621 31165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=38317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:04.204799 31165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:04.205838 31165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:04.215297 31171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.215317 31174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.215293 31170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:04.215731 31165 server_base.cc:1034] running on GCE node
I20260810 14:36:04.216431 31165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:04.217742 31165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:04.218982 31165 hybrid_clock.cc:648] HybridClock initialized: now 1786372564218955 us; error 44 us; skew 500 ppm
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372564, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:04.222204 31165 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:36:04.223300 31165 webserver.cc:460] Webserver started at http://127.21.43.126:38317/ using document root <none> and password file <none>
I20260810 14:36:04.223946 31165 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:04.224073 31165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:04.228430 31165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:36:04.230873 31182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.232290 31165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:36:04.232455 31165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "07e05a6c89a6455a8f0b5d4b788ce486"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "acfa6f95e4a8060d2000a0544b76311f"
server_key_iv: "6118cf6d4225ed35cfd8d8ae77cc810d"
server_key_version: "encryptionkey@0"
I20260810 14:36:04.233064 31165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:04.256650 31165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:04.260227 31165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:04.260480 31165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:04.268338 31165 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43339
I20260810 14:36:04.268361 31241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43339 every 8 connection(s)
I20260810 14:36:04.269479 31165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:04.270555 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31165
I20260810 14:36:04.271010 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:36443
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:43339
--webserver_port=45731
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:36:04.274708 31242 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:36:04.277632 31242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Bootstrap starting.
I20260810 14:36:04.301868 31242 log.cc:824] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:04.310837 31242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:36:04.311519 31242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Bootstrap complete.
I20260810 14:36:04.317855 31242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:04.318545 31242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e05a6c89a6455a8f0b5d4b788ce486, State: Initialized, Role: FOLLOWER
I20260810 14:36:04.319219 31242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:04.319386 31242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:04.319460 31242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:04.319566 31242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:36:04.322423 31242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:04.322824 31242 leader_election.cc:302] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e05a6c89a6455a8f0b5d4b788ce486; no voters:
I20260810 14:36:04.323268 31242 leader_election.cc:288] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:36:04.323429 31247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:36:04.323729 31247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [term 2 LEADER]: Becoming Leader. State: Replica: 07e05a6c89a6455a8f0b5d4b788ce486, State: Running, Role: LEADER
I20260810 14:36:04.324070 31247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } }
I20260810 14:36:04.324710 31242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:04.325676 31249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e05a6c89a6455a8f0b5d4b788ce486. Latest consensus state: current_term: 2 leader_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } } }
I20260810 14:36:04.325676 31248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e05a6c89a6455a8f0b5d4b788ce486" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43339 } } }
I20260810 14:36:04.325842 31249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:04.325857 31248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:04.327466 31256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:04.330864 31256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:04.331594 31256 catalog_manager.cc:1259] Loaded cluster ID: db81ec08838841fda32e57b802a61f90
I20260810 14:36:04.331658 31256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:04.333978 31256 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:04.334688 31256 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07e05a6c89a6455a8f0b5d4b788ce486: Loaded TSK: 0
I20260810 14:36:04.336004 31256 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:36:04.410193 31245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:04.410454 31245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:04.410499 31245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:04.414688 31245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:04.414760 31245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:04.414845 31245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:36:04.420308 31245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.65.keytab
--principal=oryx/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:36443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=45731
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:04.421357 31245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:04.422387 31245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:04.432426 31274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.432422 31271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.432436 31272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:04.433295 31245 server_base.cc:1034] running on GCE node
I20260810 14:36:04.433729 31245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:04.434466 31245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:04.435645 31245 hybrid_clock.cc:648] HybridClock initialized: now 1786372564435608 us; error 55 us; skew 500 ppm
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372564, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:04.438789 31245 init.cc:377] Logged in from keytab as oryx/127.21.43.65@KRBTEST.COM (short username oryx)
I20260810 14:36:04.439862 31245 webserver.cc:460] Webserver started at http://127.21.43.65:45731/ using document root <none> and password file <none>
I20260810 14:36:04.440442 31245 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:04.440558 31245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:04.444702 31245 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:04.446924 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.448323 31245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:04.448427 31245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "ebe27668b6fe4ae0942373a10767a092"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "bd9ab9e7c1f74d7ddea5d4d4ca7180a9"
server_key_iv: "98b67c7ed972cade4c2a9549f5ec7740"
server_key_version: "encryptionkey@0"
I20260810 14:36:04.449023 31245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:04.462994 31245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:04.466595 31245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:04.466830 31245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:04.467689 31245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:04.468588 31245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:04.468638 31245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.468681 31245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:04.468701 31245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.479085 31245 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:36443
I20260810 14:36:04.479104 31408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:36443 every 8 connection(s)
I20260810 14:36:04.480230 31245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:04.490258 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31245
I20260810 14:36:04.490743 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:33315
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:43339
--webserver_port=44077
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.493661 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.482421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:37783)
0810 14:36:04.483126 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.483228 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.483976 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.484960 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.484966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.485329 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.485767 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.485781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.487220 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.487224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.488195 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.488207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.488997 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.489000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.489147 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.489761 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.489778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.492843 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":151,"threads_started":1}
W20260810 14:36:04.494132 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.482353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:37783 (local address 127.21.43.126:43339)
0810 14:36:04.482836 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.482856 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.483342 (+ 486us) server_negotiation.cc:407] Connection header received
0810 14:36:04.484329 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.484339 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.484615 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.484877 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.485958 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.486975 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.488336 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.488818 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.488952 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.493179 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:37783: BlockingRecv error: recv got EOF from 127.21.43.65:37783 (error 108)
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":106,"threads_started":1}
W20260810 14:36:04.494326 31409 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:43339 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.43.126:43339: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.501255 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.494916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:54315)
0810 14:36:04.495039 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.495056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.495183 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.495455 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.495458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.495598 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.495701 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.495706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.496885 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.496888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.497434 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.497442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.498130 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.498133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.498201 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.498852 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.498864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.501121 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:36:04.501521 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.495022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:54315 (local address 127.21.43.126:43339)
0810 14:36:04.495182 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.495188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.495208 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:36:04.495268 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.495273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.495335 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.495431 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.495833 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.496761 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.497561 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.498006 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.498144 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.501354 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:54315: BlockingRecv error: recv got EOF from 127.21.43.65:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.508186 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.501960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:32975)
0810 14:36:04.502099 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.502115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.502202 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.502491 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.502493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.502624 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.502720 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.502724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.504033 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.504036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.504564 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.504572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.505207 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.505210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.505289 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.505866 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.505879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.508066 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 14:36:04.508395 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.502077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:32975 (local address 127.21.43.126:43339)
0810 14:36:04.502229 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.502234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.502249 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:04.502300 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.502302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.502362 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.502466 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.502856 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.503863 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.504687 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.505054 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.505166 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.508266 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:32975: BlockingRecv error: recv got EOF from 127.21.43.65:32975 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 14:36:04.508528 31409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:36:04.612179 31414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:04.612465 31414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:04.612511 31414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:04.616726 31414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:04.616806 31414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:04.616899 31414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:36:04.622712 31414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.66.keytab
--principal=oryx/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:33315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=44077
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:04.624042 31414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:04.625201 31414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:04.635707 31421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.635689 31420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.635743 31423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:04.636188 31414 server_base.cc:1034] running on GCE node
I20260810 14:36:04.636658 31414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:04.637458 31414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:04.638655 31414 hybrid_clock.cc:648] HybridClock initialized: now 1786372564638640 us; error 33 us; skew 500 ppm
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372564, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:04.641877 31414 init.cc:377] Logged in from keytab as oryx/127.21.43.66@KRBTEST.COM (short username oryx)
I20260810 14:36:04.642904 31414 webserver.cc:460] Webserver started at http://127.21.43.66:44077/ using document root <none> and password file <none>
I20260810 14:36:04.643590 31414 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:04.643718 31414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:04.648157 31414 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:04.650602 31431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.651978 31414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:36:04.652249 31414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "1028e9cffca5430282e6c0236b9d5309"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "ef605f5cc212026bec0823c3dc4deb37"
server_key_iv: "4fbbcbc2d24ceb9cdb69f42110eabe3c"
server_key_version: "encryptionkey@0"
I20260810 14:36:04.652881 31414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:04.680341 31414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:04.684023 31414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:04.684262 31414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:04.684996 31414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:04.685911 31414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:04.685986 31414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.686064 31414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:04.686110 31414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.695598 31414 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:33315
I20260810 14:36:04.695613 31558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:33315 every 8 connection(s)
I20260810 14:36:04.696717 31414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:04.700264 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31414
I20260810 14:36:04.700780 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:35661
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:43339
--webserver_port=39535
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:36:04.710980 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.698895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:46945 (local address 127.21.43.126:43339)
0810 14:36:04.699060 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.699064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.700286 (+ 1222us) server_negotiation.cc:407] Connection header received
0810 14:36:04.701506 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.701509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.701585 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.701708 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.703299 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.703922 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.705818 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.706114 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.706273 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.710831 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:46945: BlockingRecv error: recv got EOF from 127.21.43.66:46945 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:36:04.711715 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.699380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:46945)
0810 14:36:04.700116 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.700161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.701171 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.701908 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.701915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.702319 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.703067 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.703090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.704070 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.704077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.705635 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.705652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.706330 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.706337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.706626 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.707479 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.707510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.710670 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":139,"threads_started":1}
W20260810 14:36:04.712262 31559 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:43339 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.43.126:43339: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.718778 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.712818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:58325)
0810 14:36:04.712964 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.712981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.713116 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.713395 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.713398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.713575 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.713675 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.713680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.714447 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.714455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.714989 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.714997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.715519 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.715524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.715626 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.716162 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.716174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.718609 (+ 2435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 14:36:04.718889 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.712908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:58325 (local address 127.21.43.126:43339)
0810 14:36:04.713038 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.713042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.713059 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:36:04.713235 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.713243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.713288 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.713379 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.713797 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.714322 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.715116 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.715398 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.715515 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.718767 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:58325: BlockingRecv error: recv got EOF from 127.21.43.66:58325 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.725689 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.719593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:40725)
0810 14:36:04.719763 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.719784 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.719882 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.720131 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.720134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.720309 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.720410 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.720416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.721210 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.721213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.722114 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.722126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.722576 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.722579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.722673 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.723211 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.723226 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.725533 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 14:36:04.725907 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.719649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:40725 (local address 127.21.43.126:43339)
0810 14:36:04.719764 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.719767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.719843 (+ 76us) server_negotiation.cc:407] Connection header received
0810 14:36:04.719970 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.719972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.720011 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.720086 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.720556 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.721089 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.722242 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.722465 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.722548 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.725751 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:40725: BlockingRecv error: recv got EOF from 127.21.43.66:40725 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 14:36:04.726189 31559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:36:04.826766 31562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:04.827126 31562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:04.827270 31562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:04.832041 31562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:04.832114 31562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:04.832180 31562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:36:04.838054 31562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/krb5kdc/oryx_127.21.43.67.keytab
--principal=oryx/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:35661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=39535
--tserver_master_addrs=127.21.43.126:43339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:04.839309 31562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:04.840421 31562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:36:04.851046 31562 server_base.cc:1034] running on GCE node
W20260810 14:36:04.850978 31571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.851011 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:04.850986 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:04.851886 31562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:04.852607 31562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:04.853796 31562 hybrid_clock.cc:648] HybridClock initialized: now 1786372564853777 us; error 32 us; skew 500 ppm
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372564, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:04.856940 31562 init.cc:377] Logged in from keytab as oryx/127.21.43.67@KRBTEST.COM (short username oryx)
I20260810 14:36:04.857971 31562 webserver.cc:460] Webserver started at http://127.21.43.67:39535/ using document root <none> and password file <none>
I20260810 14:36:04.858531 31562 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:04.858619 31562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:04.862959 31562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:04.865870 31579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.867635 31562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:36:04.867789 31562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "153dee6e00064e469e35a9aeb8aee6f2"
format_stamp: "Formatted at 2026-08-10 14:36:03 on dist-test-slave-chsk"
server_key: "981ec2c81d2017fdab967f4417a16261"
server_key_iv: "5000dbed66964d6af4f4a0fa332716ba"
server_key_version: "encryptionkey@0"
I20260810 14:36:04.868402 31562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:04.890301 31562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:04.893760 31562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:04.894001 31562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:04.894713 31562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:04.895660 31562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:04.895712 31562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.895754 31562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:04.895814 31562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:04.905879 31562 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:35661
I20260810 14:36:04.905896 31710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:35661 every 8 connection(s)
I20260810 14:36:04.907064 31562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:04.910454 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31562
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:36:04.921253 31411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.909209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:41639 (local address 127.21.43.126:43339)
0810 14:36:04.909390 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.909395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.910817 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 14:36:04.911938 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.911943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.911997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.912092 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.913640 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.914217 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.915864 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.916179 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.916264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.921117 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:41639: BlockingRecv error: recv got EOF from 127.21.43.67:41639 (error 108)
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":73}
I20260810 14:36:04.922107 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.909677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:41639)
0810 14:36:04.910608 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.910675 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.911726 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.912333 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.912346 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.912862 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.913462 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.913477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.914355 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.914359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.915723 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.915735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.916364 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.916369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.916612 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.917248 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.917266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.920995 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":140,"threads_started":1}
I20260810 14:36:04.922425 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.911684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.21.43.126:43339)
0810 14:36:04.911927 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.911931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.911943 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:36:04.911969 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.911971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.911999 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.912100 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.913119 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.913765 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.914883 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.915215 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.915314 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.917250 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:04.917255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:04.917272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:04.917302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:04.919219 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:04.919787 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:04.919794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:04.919801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:04.920257 (+ 456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:04.920800 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:04.920804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:04.920805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:04.921231 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:04.921467 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:04.921899 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 14:36:04.922206 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
W20260810 14:36:04.923486 31711 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:43339 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.43.126:43339: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.930753 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.923974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:60059)
0810 14:36:04.924109 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.924124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.924218 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.924461 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.924463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.924608 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.924792 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.924800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.925731 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.925734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.926447 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.926458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.926952 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.926956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.927050 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.927647 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.927666 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.930570 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 14:36:04.930881 31715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.924091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:60059 (local address 127.21.43.126:43339)
0810 14:36:04.924232 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.924236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.924253 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:36:04.924298 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.924300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.924343 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.924420 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.924903 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.925610 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.926575 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.926816 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.926953 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.930760 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:60059: BlockingRecv error: recv got EOF from 127.21.43.67:60059 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:04 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:04.936906 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.931478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:51677)
0810 14:36:04.931609 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.931626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:04.931719 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:04.931976 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:04.931979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:04.932163 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:04.932246 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.932252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.933011 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.933016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.933557 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:04.933565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.934036 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.934040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:04.934135 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.934639 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:04.934652 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:04.936734 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 14:36:04.936987 31715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:04.931579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:51677 (local address 127.21.43.126:43339)
0810 14:36:04.931757 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:04.931763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:04.931778 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:04.931830 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:04.931832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:04.931868 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:04.931957 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:04.932371 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.932896 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.933687 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:04.933909 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:04.934020 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:04.936869 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:51677: BlockingRecv error: recv got EOF from 127.21.43.67:51677 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 14:36:04.937232 31711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:05.514981 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.509223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:46737)
0810 14:36:05.509530 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.509548 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:05.509664 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:05.509928 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:05.509931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:05.510162 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:05.510258 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.510264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.511065 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.511070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.511679 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.511688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.512170 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.512173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.512259 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.512806 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:05.512817 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:05.514844 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260810 14:36:05.515151 31717 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:05.509388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:46737 (local address 127.21.43.126:43339)
0810 14:36:05.509649 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:05.509652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:05.509667 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:05.509746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:05.509748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:05.509790 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:05.509900 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:05.510393 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.510942 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.511813 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.512046 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.512162 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.515003 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:46737: BlockingRecv error: recv got EOF from 127.21.43.65:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:05.733485 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.726823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:44103)
0810 14:36:05.727106 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.727119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:05.727277 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:05.727526 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:05.727528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:05.727715 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:05.727813 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.727818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.729106 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.729110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.729690 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.729699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.730363 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.730366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.730486 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.731062 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:05.731087 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:05.733319 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
W20260810 14:36:05.733783 31717 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:05.726961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:44103 (local address 127.21.43.126:43339)
0810 14:36:05.727123 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:05.727126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:05.727234 (+ 108us) server_negotiation.cc:407] Connection header received
0810 14:36:05.727362 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:05.727365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:05.727410 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:05.727491 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:05.727993 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.728953 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.729823 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.730110 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.730212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.733623 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:44103: BlockingRecv error: recv got EOF from 127.21.43.66:44103 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:05 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:05.943329 31719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.937933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:56789)
0810 14:36:05.938258 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.938271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:05.938374 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:05.938609 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:05.938611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:05.938798 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:05.938895 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.938901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.939769 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.939771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.940331 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:05.940339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.940777 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.940779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:05.940837 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.941348 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:05.941360 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:05.943189 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
W20260810 14:36:05.943653 31717 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:05.937981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:56789 (local address 127.21.43.126:43339)
0810 14:36:05.938137 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:05.938142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:05.938366 (+ 224us) server_negotiation.cc:407] Connection header received
0810 14:36:05.938457 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:05.938459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:05.938508 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:05.938571 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:05.939049 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.939660 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.940441 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:05.940673 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:05.940756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:05.943425 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:56789: BlockingRecv error: recv got EOF from 127.21.43.67:56789 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:06.521245 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.515797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:43995)
0810 14:36:06.516119 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.516133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:06.516225 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:06.516474 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:06.516475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:06.516658 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:06.516754 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.516759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.517459 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.517461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.518009 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.518026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.518639 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.518642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.518711 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.519268 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:06.519280 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:06.521097 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260810 14:36:06.521472 31721 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:06.515909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:43995 (local address 127.21.43.126:43339)
0810 14:36:06.516162 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:06.516166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:06.516180 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:36:06.516317 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:06.516319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:06.516363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:06.516438 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:06.516852 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.517368 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.518293 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.518507 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.518581 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.521232 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:43995: BlockingRecv error: recv got EOF from 127.21.43.65:43995 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:06.740167 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.734494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:58319)
0810 14:36:06.734776 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.734791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:06.734928 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:06.735265 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:06.735269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:06.735523 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:06.735631 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.735637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.736394 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.736399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.737037 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.737047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.737451 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.737454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.737567 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.738141 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:06.738162 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:06.740017 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
W20260810 14:36:06.740338 31721 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:06.734568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:58319 (local address 127.21.43.126:43339)
0810 14:36:06.734719 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:06.734724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:06.734876 (+ 152us) server_negotiation.cc:407] Connection header received
0810 14:36:06.735040 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:06.735044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:06.735099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:06.735204 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:06.735738 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.736260 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.737150 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.737360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.737432 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.740167 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:58319: BlockingRecv error: recv got EOF from 127.21.43.66:58319 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:06 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:06.954348 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.944175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:36007)
0810 14:36:06.944488 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.944502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:06.944603 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:06.944847 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:06.944849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:06.945053 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:06.945151 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.945157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.946167 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.946170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.946866 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:06.946876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.947398 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.947401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:06.947470 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.952112 (+ 4642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:06.952127 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:06.954189 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
W20260810 14:36:06.954605 31721 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:06.944354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:36007 (local address 127.21.43.126:43339)
0810 14:36:06.944502 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:06.944505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:06.944570 (+ 65us) server_negotiation.cc:407] Connection header received
0810 14:36:06.944683 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:06.944685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:06.944733 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:06.944809 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:06.945278 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.946033 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.946977 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:06.947254 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:06.947350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:06.954373 (+ 7023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:36007: BlockingRecv error: recv got EOF from 127.21.43.67:36007 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:07.528182 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.522218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:57753)
0810 14:36:07.522635 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:07.522651 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:07.522800 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:07.523172 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:07.523175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:07.523426 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:07.523575 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.523588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.524361 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.524363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.524996 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.525005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.525538 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.525541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.525603 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.526189 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:07.526201 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:07.528065 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":88,"threads_started":1}
W20260810 14:36:07.528345 31726 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:07.522321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:57753 (local address 127.21.43.126:43339)
0810 14:36:07.522626 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.522630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.522738 (+ 108us) server_negotiation.cc:407] Connection header received
0810 14:36:07.522929 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.522932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.522987 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.523096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.523714 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.524266 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.525190 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.525414 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.525496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.528209 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:57753: BlockingRecv error: recv got EOF from 127.21.43.65:57753 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:07.746688 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.740943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:57483)
0810 14:36:07.741314 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:07.741329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:07.741468 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:07.741752 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:07.741755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:07.741975 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:07.742083 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.742089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.742876 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.742880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.743534 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.743543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.744003 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.744006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.744092 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.744655 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:07.744668 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:07.746508 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
W20260810 14:36:07.746755 31726 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:07.741085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:57483 (local address 127.21.43.126:43339)
0810 14:36:07.741280 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.741285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.741411 (+ 126us) server_negotiation.cc:407] Connection header received
0810 14:36:07.741574 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.741577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.741638 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.741728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.742215 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.742758 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.743660 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.743884 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.743998 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.746644 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:57483: BlockingRecv error: recv got EOF from 127.21.43.66:57483 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:07 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:07.962118 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.955397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:55699)
0810 14:36:07.955720 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:07.955733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:07.955876 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:07.956250 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:07.956252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:07.956444 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:07.956643 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.956652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.957528 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.957532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.958364 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:07.958375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.958851 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.958854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:07.958919 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.959531 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:07.959543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:07.961958 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
W20260810 14:36:07.962284 31726 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:07.955511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:55699 (local address 127.21.43.126:43339)
0810 14:36:07.955657 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.955662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.955834 (+ 172us) server_negotiation.cc:407] Connection header received
0810 14:36:07.956010 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.956016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.956086 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.956194 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.956764 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.957396 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.958489 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.958715 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.958806 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.962152 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:55699: BlockingRecv error: recv got EOF from 127.21.43.67:55699 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:08.534909 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:08.529114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:35575)
0810 14:36:08.529517 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:08.529535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:08.529651 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:08.529955 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:08.529957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:08.530148 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:08.530247 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.530252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.530969 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.530971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.531534 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.531542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.532011 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.532013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.532073 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.532564 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:08.532576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:08.534789 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":88,"threads_started":1}
W20260810 14:36:08.535228 31730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:08.529209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:35575 (local address 127.21.43.126:43339)
0810 14:36:08.529591 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:08.529594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:08.529606 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:36:08.529778 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:08.529781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:08.529838 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:08.529913 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:08.530351 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.530878 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.531657 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.531902 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.531992 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.534932 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:35575: BlockingRecv error: recv got EOF from 127.21.43.65:35575 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":96,"threads_started":1}
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:08.753351 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:08.747630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:60621)
0810 14:36:08.747926 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:08.747940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:08.748053 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:08.748303 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:08.748305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:08.748508 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:08.748610 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.748616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.749398 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.749403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.750020 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.750034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.750587 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.750590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.750685 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.751259 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:08.751271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:08.753225 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
W20260810 14:36:08.753650 31730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:08.747730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:60621 (local address 127.21.43.126:43339)
0810 14:36:08.747881 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:08.747884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:08.748014 (+ 130us) server_negotiation.cc:407] Connection header received
0810 14:36:08.748141 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:08.748143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:08.748189 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:08.748266 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:08.748714 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.749269 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.750142 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.750454 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.750575 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.753444 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:60621: BlockingRecv error: recv got EOF from 127.21.43.66:60621 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:08 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:08.968684 31733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:08.962907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:45505)
0810 14:36:08.963312 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:08.963325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:08.963453 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:08.963747 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:08.963750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:08.963968 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:08.964063 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.964069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.964848 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.964853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.965413 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:08.965421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.965897 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.965901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:08.965988 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.966494 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:08.966505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:08.968488 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":154,"threads_started":1}
W20260810 14:36:08.968731 31730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:08.963036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45505 (local address 127.21.43.126:43339)
0810 14:36:08.963237 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:08.963243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:08.963400 (+ 157us) server_negotiation.cc:407] Connection header received
0810 14:36:08.963573 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:08.963576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:08.963633 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:08.963718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:08.964194 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.964733 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.965539 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:08.965777 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:08.965890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:08.968626 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:45505: BlockingRecv error: recv got EOF from 127.21.43.67:45505 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:09.542148 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:09.535856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:58605)
0810 14:36:09.536213 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:09.536227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:09.536332 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:09.536628 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:09.536630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:09.536835 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:09.537028 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.537041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.538166 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.538173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.538853 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.538863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.539340 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.539343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.539413 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.540024 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:09.540056 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:09.541952 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
W20260810 14:36:09.542232 31735 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:09.535938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:58605 (local address 127.21.43.126:43339)
0810 14:36:09.536279 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:09.536285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:09.536304 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:36:09.536420 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:09.536422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:09.536466 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:09.536550 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:09.537220 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.538024 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.538956 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.539208 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.539293 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.542097 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:58605: BlockingRecv error: recv got EOF from 127.21.43.65:58605 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":149,"threads_started":1}
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:09.760272 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:09.754174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:34295)
0810 14:36:09.754539 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:09.754557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:09.754705 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:09.754977 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:09.754980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:09.755282 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:09.755382 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.755389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.756223 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.756226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.756837 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.756848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.757338 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.757342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.757412 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.757954 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:09.757966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:09.760072 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
W20260810 14:36:09.760329 31735 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:09.754369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:34295 (local address 127.21.43.126:43339)
0810 14:36:09.754547 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:09.754553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:09.754643 (+ 90us) server_negotiation.cc:407] Connection header received
0810 14:36:09.754822 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:09.754824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:09.754869 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:09.754956 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:09.755501 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.756030 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.756943 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.757162 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.757293 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.760217 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:34295: BlockingRecv error: recv got EOF from 127.21.43.66:34295 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:09 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:09.975649 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:09.969614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:52783)
0810 14:36:09.969974 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:09.969992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:09.970171 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:09.970462 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:09.970465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:09.970699 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:09.970794 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.970800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.971689 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.971694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.972337 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:09.972346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.972838 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.972842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:09.972930 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.973499 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:09.973513 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:09.975462 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
W20260810 14:36:09.975718 31735 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:09.969816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:52783 (local address 127.21.43.126:43339)
0810 14:36:09.969974 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:09.969980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:09.970106 (+ 126us) server_negotiation.cc:407] Connection header received
0810 14:36:09.970283 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:09.970285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:09.970344 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:09.970436 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:09.970922 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.971574 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.972469 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:09.972711 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:09.972821 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:09.975604 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:52783: BlockingRecv error: recv got EOF from 127.21.43.67:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:10.549055 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.543200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:52781)
0810 14:36:10.543519 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:10.543534 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:10.543639 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:10.544017 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:10.544020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:10.544292 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:10.544395 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.544401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.545325 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.545328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.545940 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.545950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.546414 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.546416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.546481 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.546988 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:10.547000 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:10.548929 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
W20260810 14:36:10.549235 31740 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:10.543349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:52781 (local address 127.21.43.126:43339)
0810 14:36:10.543714 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:10.543717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:10.543742 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:36:10.543788 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:10.543790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:10.543834 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:10.543928 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:10.544527 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.545205 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.546050 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.546305 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.546389 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.549092 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:52781: BlockingRecv error: recv got EOF from 127.21.43.65:52781 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":144,"threads_started":1}
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:10.767923 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.761283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:43435)
0810 14:36:10.761680 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:10.761700 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:10.761881 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:10.762198 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:10.762201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:10.762443 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:10.762543 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.762548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.763500 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.763504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.764247 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.764264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.764919 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.764923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.764999 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.765542 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:10.765555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:10.767778 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":117,"threads_started":1}
W20260810 14:36:10.768157 31740 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:10.761469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:43435 (local address 127.21.43.126:43339)
0810 14:36:10.761686 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:10.761693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:10.761801 (+ 108us) server_negotiation.cc:407] Connection header received
0810 14:36:10.762007 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:10.762011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:10.762070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:10.762166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:10.762677 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.763328 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.764406 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.764746 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.764836 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.767980 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:43435: BlockingRecv error: recv got EOF from 127.21.43.66:43435 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:10 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:10.982432 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.976676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:56943)
0810 14:36:10.977009 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:10.977023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:10.977135 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:10.977407 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:10.977410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:10.977622 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:10.977724 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.977729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.978541 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.978544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.979128 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.979177 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.979666 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.979668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.979732 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.980280 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:10.980291 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:10.982294 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
W20260810 14:36:10.982642 31740 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:10.976826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:56943 (local address 127.21.43.126:43339)
0810 14:36:10.976978 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:10.976982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:10.977133 (+ 151us) server_negotiation.cc:407] Connection header received
0810 14:36:10.977209 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:10.977211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:10.977274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:10.977355 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:10.977847 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.978397 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.979326 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.979553 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.979647 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.982492 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:56943: BlockingRecv error: recv got EOF from 127.21.43.67:56943 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:11.556437 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.550069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:38949)
0810 14:36:11.550435 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.550450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.550546 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.550841 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.550843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.551035 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:11.551191 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.551203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.552169 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.552173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.552899 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.552910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.553464 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.553467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.553617 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.554245 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:11.554258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:11.556309 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
W20260810 14:36:11.556643 31744 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:11.550203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:38949 (local address 127.21.43.126:43339)
0810 14:36:11.550499 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.550503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.550520 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:36:11.550642 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.550645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.550691 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.550790 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.551317 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.552058 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.553030 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.553282 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.553442 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.556479 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:38949: BlockingRecv error: recv got EOF from 127.21.43.65:38949 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:11.774865 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.768895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:48013)
0810 14:36:11.769270 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.769289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.769433 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.769726 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.769730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.769983 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:11.770143 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.770152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.770951 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.770954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.771580 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.771589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.772071 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.772076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.772158 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.772754 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:11.772766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:11.774737 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
W20260810 14:36:11.775020 31744 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:11.768987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48013 (local address 127.21.43.126:43339)
0810 14:36:11.769160 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.769165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.769402 (+ 237us) server_negotiation.cc:407] Connection header received
0810 14:36:11.769516 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.769519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.769580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.769663 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.770279 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.770825 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.771693 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.771922 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.772004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.774896 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:48013: BlockingRecv error: recv got EOF from 127.21.43.66:48013 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:11 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:11.994158 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.983418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:46063)
0810 14:36:11.983765 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.983779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.983892 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.984215 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.984218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.984420 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:11.984524 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.984529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.985327 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.985331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.985907 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.985916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.986386 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.986389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.986463 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.991848 (+ 5385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:11.991863 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:11.994007 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
W20260810 14:36:11.994414 31744 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:11.983609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:46063 (local address 127.21.43.126:43339)
0810 14:36:11.983762 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.983766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.983844 (+ 78us) server_negotiation.cc:407] Connection header received
0810 14:36:11.983977 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.983980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.984044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.984128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.984642 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.985187 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.986036 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.986251 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.986333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.994272 (+ 7939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:46063: BlockingRecv error: recv got EOF from 127.21.43.67:46063 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:12.563464 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.557550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:57905)
0810 14:36:12.557884 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:12.557899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:12.557994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:12.558287 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:12.558289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:12.558489 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:12.558590 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.558595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.559490 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.559493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.560072 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.560081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.560535 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.560537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.560606 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.561243 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:12.561256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:12.563274 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":137,"threads_started":1}
W20260810 14:36:12.563781 31748 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:12.557700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:57905 (local address 127.21.43.126:43339)
0810 14:36:12.557968 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.557971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.557987 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:36:12.558111 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.558113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.558162 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.558251 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.558735 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.559361 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.560197 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.560425 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.560515 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.563485 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:57905: BlockingRecv error: recv got EOF from 127.21.43.65:57905 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:12.781769 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.775857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:49881)
0810 14:36:12.776157 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:12.776170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:12.776268 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:12.776516 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:12.776518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:12.776707 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:12.776806 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.776811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.777725 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.777727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.778287 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.778295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.778829 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.778832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.778917 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.779557 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:12.779570 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:12.781634 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
W20260810 14:36:12.782054 31748 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:12.775929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:49881 (local address 127.21.43.126:43339)
0810 14:36:12.776127 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.776131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.776244 (+ 113us) server_negotiation.cc:407] Connection header received
0810 14:36:12.776351 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.776354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.776399 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.776476 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.776925 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.777625 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.778433 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.778678 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.778772 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.781876 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:49881: BlockingRecv error: recv got EOF from 127.21.43.66:49881 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:12 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:13.001734 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.995306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:43957)
0810 14:36:12.995652 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:12.995670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:12.995844 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:12.996152 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:12.996155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:12.996378 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:12.996474 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.996480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.997301 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.997304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.997910 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.997920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.998634 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.998638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.998740 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.999442 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:12.999463 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:13.001601 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":96,"threads_started":1}
W20260810 14:36:13.002054 31748 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:12.995418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:43957 (local address 127.21.43.126:43339)
0810 14:36:12.995648 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.995653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.995775 (+ 122us) server_negotiation.cc:407] Connection header received
0810 14:36:12.995958 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.995963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.996016 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.996110 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.996603 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.997157 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.998120 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.998455 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.998573 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.001895 (+ 3322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:43957: BlockingRecv error: recv got EOF from 127.21.43.67:43957 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 14:36:13 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:13 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:13.570489 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:13.564436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:51675)
0810 14:36:13.564757 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:13.564772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:13.564872 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:13.565126 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:13.565128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:13.565316 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:13.565415 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.565419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.566194 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.566197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.566752 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.566760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.567315 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.567321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.567438 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.567981 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:13.567994 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:13.570317 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
W20260810 14:36:13.570593 31752 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:13.564509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:51675 (local address 127.21.43.126:43339)
0810 14:36:13.564804 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:13.564808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:13.564824 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:36:13.564960 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:13.564962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:13.565005 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:13.565090 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:13.565521 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.566086 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.566869 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.567100 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.567261 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.570431 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:51675: BlockingRecv error: recv got EOF from 127.21.43.65:51675 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":106,"threads_started":1}
Aug 10 14:36:13 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:13 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:13.789299 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:13.782911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:55031)
0810 14:36:13.783353 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:13.783370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:13.783531 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:13.783835 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:13.783837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:13.784066 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:13.784171 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.784177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.785056 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.785061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.785641 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.785649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.786134 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.786137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.786228 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.786897 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:13.786910 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:13.789052 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":167,"threads_started":1}
W20260810 14:36:13.789417 31752 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:13.783075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:55031 (local address 127.21.43.126:43339)
0810 14:36:13.783320 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:13.783325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:13.783466 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:36:13.783647 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:13.783650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:13.783707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:13.783808 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:13.784300 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.784940 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.785768 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.786004 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.786120 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.789291 (+ 3171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:55031: BlockingRecv error: recv got EOF from 127.21.43.66:55031 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:14.009013 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.002648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:48721)
0810 14:36:14.003040 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.003063 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.003244 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.003670 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.003673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.003933 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:14.004034 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.004040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.004854 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.004859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.005428 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.005436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.005931 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.005935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.006023 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.006658 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:14.006672 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:14.008796 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":167,"threads_started":1}
W20260810 14:36:14.009063 31752 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:14.002839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:48721 (local address 127.21.43.126:43339)
0810 14:36:14.003002 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.003008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.003176 (+ 168us) server_negotiation.cc:407] Connection header received
0810 14:36:14.003481 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.003484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.003551 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.003643 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.004166 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.004736 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.005558 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.005805 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.005921 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.008945 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:48721: BlockingRecv error: recv got EOF from 127.21.43.67:48721 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:14.577452 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.571404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:56703)
0810 14:36:14.571737 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.571756 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.571867 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.572119 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.572121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.572309 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:14.572407 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.572412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.573236 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.573239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.573794 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.573802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.574414 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.574420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.574517 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.575215 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:14.575236 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:14.577274 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
W20260810 14:36:14.577543 31756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:14.571552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:56703 (local address 127.21.43.126:43339)
0810 14:36:14.571852 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.571856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.571871 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:14.571940 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.571943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.571986 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.572082 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.572510 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.573132 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.573913 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.574202 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.574321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.577405 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:56703: BlockingRecv error: recv got EOF from 127.21.43.65:56703 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":144,"threads_started":1}
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:14 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:14.796064 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.790192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:42835)
0810 14:36:14.790538 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.790554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.790657 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.790889 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.790891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.791104 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:14.791278 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.791286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.792011 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.792014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.792717 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.792728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.793160 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.793163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.793233 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.793756 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:14.793769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:14.795908 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":128,"threads_started":1}
W20260810 14:36:14.796216 31756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:14.790251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:42835 (local address 127.21.43.126:43339)
0810 14:36:14.790392 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.790396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.790622 (+ 226us) server_negotiation.cc:407] Connection header received
0810 14:36:14.790735 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.790737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.790778 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.790844 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.791394 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.791909 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.792835 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.793053 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.793128 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.796077 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:42835: BlockingRecv error: recv got EOF from 127.21.43.66:42835 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:15.015882 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:15.010004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:49343)
0810 14:36:15.010333 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:15.010351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:15.010464 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:15.010770 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:15.010773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:15.010991 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:15.011084 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.011089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.011926 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.011931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.012549 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.012558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.013023 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.013026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.013113 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.013727 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:15.013739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:15.015721 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
W20260810 14:36:15.016029 31756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:15.010246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:49343 (local address 127.21.43.126:43339)
0810 14:36:15.010420 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:15.010426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:15.010441 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:15.010585 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:15.010588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:15.010651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:15.010740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:15.011265 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.011810 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.012675 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.012902 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.013015 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.015892 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:49343: BlockingRecv error: recv got EOF from 127.21.43.67:49343 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:15.584776 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:15.578543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:46823)
0810 14:36:15.578859 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:15.578874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:15.578976 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:15.579312 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:15.579314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:15.579513 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:15.579617 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.579622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.580425 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.580429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.580986 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.580995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.581552 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.581556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.581642 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.582288 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:15.582306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:15.584645 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
W20260810 14:36:15.585148 31761 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:15.578702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:46823 (local address 127.21.43.126:43339)
0810 14:36:15.578958 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:15.578961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:15.578977 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:36:15.579099 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:15.579102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:15.579173 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:15.579279 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:15.579727 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.580284 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.581144 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.581380 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.581496 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.584843 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:46823: BlockingRecv error: recv got EOF from 127.21.43.65:46823 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:15 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:15.803292 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:15.797079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:42567)
0810 14:36:15.797397 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:15.797416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:15.797524 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:15.797793 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:15.797796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:15.798010 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:15.798115 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.798121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.798854 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.798856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.799502 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:15.799512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.800029 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.800032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:15.800106 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.800717 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:15.800736 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:15.803085 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":130,"threads_started":1}
W20260810 14:36:15.803395 31761 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:15.797224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:42567 (local address 127.21.43.126:43339)
0810 14:36:15.797402 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:15.797406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:15.797473 (+ 67us) server_negotiation.cc:407] Connection header received
0810 14:36:15.797619 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:15.797622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:15.797669 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:15.797752 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:15.798235 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.798757 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.799633 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.799870 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.799969 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.803270 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:42567: BlockingRecv error: recv got EOF from 127.21.43.66:42567 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:16.022967 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:16.017136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:52977)
0810 14:36:16.017455 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:16.017472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:16.017644 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:16.017936 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:16.017938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:16.018160 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:16.018262 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.018268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.019041 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.019046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.019692 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.019702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.020209 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.020213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.020302 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.020871 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:16.020882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:16.022780 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
W20260810 14:36:16.023030 31761 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:16.017297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:52977 (local address 127.21.43.126:43339)
0810 14:36:16.017455 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:16.017460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:16.017576 (+ 116us) server_negotiation.cc:407] Connection header received
0810 14:36:16.017760 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:16.017763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:16.017819 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:16.017910 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:16.018388 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.018923 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.019843 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.020078 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.020194 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.022918 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:52977: BlockingRecv error: recv got EOF from 127.21.43.67:52977 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:16.591679 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:16.585794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:57731)
0810 14:36:16.586136 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:16.586150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:16.586254 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:16.586507 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:16.586509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:16.586705 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:16.586802 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.586807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.587737 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.587741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.588553 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.588564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.589130 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.589133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.589195 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.589707 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:16.589719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:16.591558 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
W20260810 14:36:16.591842 31765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:16.585948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:57731 (local address 127.21.43.126:43339)
0810 14:36:16.586263 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:16.586267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:16.586304 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:36:16.586346 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:16.586348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:16.586393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:16.586466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:16.586939 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.587603 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.588690 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.589015 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.589105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.591707 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:57731: BlockingRecv error: recv got EOF from 127.21.43.65:57731 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:16 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:16.809885 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:16.804209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:54135)
0810 14:36:16.804509 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:16.804521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:16.804654 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:16.804930 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:16.804933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:16.805150 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:16.805252 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.805258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.806020 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.806024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.806584 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:16.806592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.807071 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.807075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:16.807193 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.807761 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:16.807773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:16.809706 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260810 14:36:16.809959 31765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:16.804258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:54135 (local address 127.21.43.126:43339)
0810 14:36:16.804424 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:16.804428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:16.804593 (+ 165us) server_negotiation.cc:407] Connection header received
0810 14:36:16.804763 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:16.804765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:16.804820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:16.804903 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:16.805367 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.805910 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.806713 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:16.806949 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:16.807056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:16.809848 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:54135: BlockingRecv error: recv got EOF from 127.21.43.66:54135 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:17.035326 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.024018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:45681)
0810 14:36:17.024374 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:17.024388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:17.024498 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:17.024786 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:17.024788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:17.024999 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:17.025099 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.025104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.025952 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.025957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.026518 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.026526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.027035 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.027038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.027125 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.033159 (+ 6034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:17.033175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:17.035132 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
W20260810 14:36:17.035560 31765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:17.024098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:45681 (local address 127.21.43.126:43339)
0810 14:36:17.024269 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:17.024274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:17.024466 (+ 192us) server_negotiation.cc:407] Connection header received
0810 14:36:17.024617 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:17.024619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:17.024677 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:17.024743 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:17.025235 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.025837 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.026680 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.026910 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.027025 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.035441 (+ 8416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:45681: BlockingRecv error: recv got EOF from 127.21.43.67:45681 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:17.598932 31769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.592712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:51597)
0810 14:36:17.593066 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:17.593081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:17.593207 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:17.593461 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:17.593463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:17.593657 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:17.593791 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.593799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.594849 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.594852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.595661 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.595672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.596162 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.596164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.596230 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.596764 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:17.596776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:17.598791 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":127,"threads_started":1}
W20260810 14:36:17.599201 31770 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:17.592854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:51597 (local address 127.21.43.126:43339)
0810 14:36:17.593100 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:17.593103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:17.593152 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:36:17.593280 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:17.593283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:17.593325 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:17.593420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:17.593981 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.594692 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.595798 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.596041 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.596142 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.598989 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:51597: BlockingRecv error: recv got EOF from 127.21.43.65:51597 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:17 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:17.817302 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.810909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:35485)
0810 14:36:17.811337 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:17.811353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:17.811458 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:17.811775 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:17.811778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:17.811985 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:17.812091 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.812097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.812956 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.812959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.813553 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.813562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.814072 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.814076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.814164 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.814750 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:17.814762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:17.817098 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
W20260810 14:36:17.817363 31770 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:17.811195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:35485 (local address 127.21.43.126:43339)
0810 14:36:17.811407 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:17.811411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:17.811429 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:17.811569 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:17.811571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:17.811619 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:17.811698 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:17.812274 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.812819 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.813696 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.813930 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.814024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.817243 (+ 3219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:35485: BlockingRecv error: recv got EOF from 127.21.43.66:35485 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:18.042451 31772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.036279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:33349)
0810 14:36:18.036561 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.036575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.036678 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.036923 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.036925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.037190 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:18.037291 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.037297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.038316 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.038319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.038982 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.038992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.039513 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.039516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.039580 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.040091 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:18.040103 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:18.042315 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260810 14:36:18.042682 31770 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:18.036425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:33349 (local address 127.21.43.126:43339)
0810 14:36:18.036561 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.036564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.036646 (+ 82us) server_negotiation.cc:407] Connection header received
0810 14:36:18.036758 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.036760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.036808 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.036892 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.037505 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.038182 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.039113 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.039395 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.039487 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.042521 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:33349: BlockingRecv error: recv got EOF from 127.21.43.67:33349 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:18.605873 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.599905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:33815)
0810 14:36:18.600224 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.600245 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.600380 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.600644 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.600646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.600840 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:18.600936 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.600941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.601734 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.601737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.602336 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.602344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.602827 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.602829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.602895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.603545 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:18.603558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:18.605742 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
W20260810 14:36:18.606029 31774 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:18.600064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:33815 (local address 127.21.43.126:43339)
0810 14:36:18.600424 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.600427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.600441 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:36:18.600479 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.600481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.600522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.600604 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.601065 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.601615 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.602474 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.602709 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.602798 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.605891 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:33815: BlockingRecv error: recv got EOF from 127.21.43.65:33815 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":102,"threads_started":1}
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:18 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:18.823948 31776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.818278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:53957)
0810 14:36:18.818553 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.818566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.818666 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.818903 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.818905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.819092 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:18.819265 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.819271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.819981 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.819983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.820551 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.820558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.821018 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.821020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.821080 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.821711 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:18.821723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:18.823751 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
W20260810 14:36:18.824007 31774 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:18.818419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:53957 (local address 127.21.43.126:43339)
0810 14:36:18.818569 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.818573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.818632 (+ 59us) server_negotiation.cc:407] Connection header received
0810 14:36:18.818748 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.818751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.818795 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.818866 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.819367 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.819890 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.820679 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.820900 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.821007 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.823887 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:53957: BlockingRecv error: recv got EOF from 127.21.43.66:53957 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.67@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:19.048929 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.043391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.67:52055)
0810 14:36:19.043675 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.043698 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:19.043831 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:19.044111 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:19.044114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:19.044346 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:19.044441 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.044446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.045212 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.045216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.045801 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.045809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.046272 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.046276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.046363 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.046884 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:19.046895 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:19.048727 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
W20260810 14:36:19.048970 31774 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:19.043448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:52055 (local address 127.21.43.126:43339)
0810 14:36:19.043610 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.043615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.043777 (+ 162us) server_negotiation.cc:407] Connection header received
0810 14:36:19.043942 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.043945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.044001 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.044086 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.044561 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.045091 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.045924 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.046151 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.046269 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.048858 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.67:52055: BlockingRecv error: recv got EOF from 127.21.43.67:52055 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.65@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:19.612208 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.606757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.65:49125)
0810 14:36:19.607067 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.607081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:19.607238 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:19.607551 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:19.607553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:19.607747 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:19.607859 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.607864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.608605 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.608607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.609179 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.609187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.609611 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.609613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.609676 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.610197 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:19.610209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:19.612066 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
W20260810 14:36:19.612362 31780 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:19.606887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:49125 (local address 127.21.43.126:43339)
0810 14:36:19.607123 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.607126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.607159 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:36:19.607366 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.607369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.607417 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.607515 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.607964 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.608501 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.609283 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.609502 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.609587 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.612221 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.65:49125: BlockingRecv error: recv got EOF from 127.21.43.65:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:36:19 dist-test-slave-chsk krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.43.66@KRBTEST.COM for oryx/127.21.43.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:36:19.830935 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.825125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43339 (local address 127.21.43.66:47987)
0810 14:36:19.825404 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.825418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:19.825521 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:19.825758 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:19.825760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:19.825947 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:19.826135 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.826142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.826879 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.826882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.827538 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.827548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.828018 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.828020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.828081 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.828627 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:19.828639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:19.830789 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.43.126:43339: Server oryx/127.21.43.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
W20260810 14:36:19.831197 31780 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:19.825220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:47987 (local address 127.21.43.126:43339)
0810 14:36:19.825412 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.825417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.825492 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:36:19.825602 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.825604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.825649 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.825723 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.826260 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.826784 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.827671 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.827907 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.828009 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.831015 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.43.66:47987: BlockingRecv error: recv got EOF from 127.21.43.66:47987 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 14:36:19.915534 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31245
I20260810 14:36:19.926167 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31414
I20260810 14:36:19.935729 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31562
I20260810 14:36:19.949019 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31165
2026-08-10T14:36:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18170 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T14:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:19Z Disabled control of system clock
I20260810 14:36:19.984884 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37191
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:36:20.102600 31786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.102900 31786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.102936 31786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.107081 31786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:20.107196 31786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.107215 31786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.107234 31786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:20.107249 31786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:20.112370 31786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:37191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:20.113571 31786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.114964 31786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.124091 31795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.124085 31793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.124168 31786 server_base.cc:1034] running on GCE node
W20260810 14:36:20.124085 31792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.124954 31786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.126169 31786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.127369 31786 hybrid_clock.cc:648] HybridClock initialized: now 1786372580127356 us; error 30 us; skew 500 ppm
I20260810 14:36:20.129438 31786 webserver.cc:460] Webserver started at http://127.21.43.126:42377/ using document root <none> and password file <none>
I20260810 14:36:20.130081 31786 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.130184 31786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.130436 31786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.132716 31786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "f2296df664874275818ab1fbaf859bcd"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "3d2cd89ccf3331ace8f369cb8968bbdd"
server_key_iv: "539e85cf3b6be7be2fb514323991e8c0"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.133356 31786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "f2296df664874275818ab1fbaf859bcd"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "3d2cd89ccf3331ace8f369cb8968bbdd"
server_key_iv: "539e85cf3b6be7be2fb514323991e8c0"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.137343 31786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:20.139978 31801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.141264 31786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:36:20.141440 31786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "f2296df664874275818ab1fbaf859bcd"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "3d2cd89ccf3331ace8f369cb8968bbdd"
server_key_iv: "539e85cf3b6be7be2fb514323991e8c0"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.141568 31786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.165228 31786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.166064 31786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.166301 31786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.174032 31786 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:37191
I20260810 14:36:20.174090 31861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:37191 every 8 connection(s)
I20260810 14:36:20.175199 31786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:20.178629 31862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:20.182155 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31786
I20260810 14:36:20.182312 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:36:20.182588 21677 external_mini_cluster.cc:1428] Setting key 1706f2b6e5191b86c2d943e1a34291f7
I20260810 14:36:20.185374 31862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: Bootstrap starting.
I20260810 14:36:20.188371 31862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:20.189221 31862 log.cc:824] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:20.191782 31862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: No bootstrap required, opened a new log
I20260810 14:36:20.193284 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.184014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.21.43.126:37191)
0810 14:36:20.184747 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.184770 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.184815 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:36:20.185685 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.185708 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.186233 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.186616 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.187126 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.188628 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.189586 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.190165 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.190332 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.191306 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.191313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.191329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.191842 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.191915 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.191942 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.192054 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.192193 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.192548 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":142,"threads_started":1}
I20260810 14:36:20.195513 31862 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } }
I20260810 14:36:20.195823 31862 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:20.195875 31862 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2296df664874275818ab1fbaf859bcd, State: Initialized, Role: FOLLOWER
I20260810 14:36:20.196362 31862 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } }
I20260810 14:36:20.196493 31862 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:20.196535 31862 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:20.196614 31862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:20.197570 31862 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } }
I20260810 14:36:20.197878 31862 leader_election.cc:302] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2296df664874275818ab1fbaf859bcd; no voters:
I20260810 14:36:20.198145 31862 leader_election.cc:288] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:20.198293 31868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:20.198571 31868 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [term 1 LEADER]: Becoming Leader. State: Replica: f2296df664874275818ab1fbaf859bcd, State: Running, Role: LEADER
I20260810 14:36:20.198974 31868 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } }
I20260810 14:36:20.199616 31862 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:20.201249 31870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2296df664874275818ab1fbaf859bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } } }
I20260810 14:36:20.201527 31870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:20.201550 31871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2296df664874275818ab1fbaf859bcd. Latest consensus state: current_term: 1 leader_uuid: "f2296df664874275818ab1fbaf859bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2296df664874275818ab1fbaf859bcd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37191 } } }
I20260810 14:36:20.201680 31871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd [sys.catalog]: This master's current role is: LEADER
W20260810 14:36:20.205612 31885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:36:20.205698 31885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:36:20.205783 31884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:20.206496 31884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:20.212407 31884 catalog_manager.cc:1347] Generated new cluster ID: 73e32062c0294fc480534e0b7d2beb18
I20260810 14:36:20.212493 31884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:20.235574 31884 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:20.236577 31884 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:20.246134 31884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2296df664874275818ab1fbaf859bcd: Generated new TSK 0
I20260810 14:36:20.246985 31884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:20.259117 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--builtin_ntp_servers=127.21.43.84:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 14:36:20.382437 31890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.382794 31890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.382834 31890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.386901 31890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.386965 31890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.387025 31890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:36:20.391778 31890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:20.392863 31890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.394270 31890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.404456 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.404457 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.404551 31890 server_base.cc:1034] running on GCE node
W20260810 14:36:20.404690 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.405324 31890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.406026 31890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.407229 31890 hybrid_clock.cc:648] HybridClock initialized: now 1786372580407215 us; error 32 us; skew 500 ppm
I20260810 14:36:20.409256 31890 webserver.cc:460] Webserver started at http://127.21.43.65:37093/ using document root <none> and password file <none>
I20260810 14:36:20.409888 31890 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.409989 31890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.410243 31890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.412436 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "48f32628127648b080afa57eae74ff64"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "ddec7ee632f5c9093b44b4440f253c97"
server_key_iv: "6c51d6a35f44c80a05eaea843664dea6"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.413043 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "48f32628127648b080afa57eae74ff64"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "ddec7ee632f5c9093b44b4440f253c97"
server_key_iv: "6c51d6a35f44c80a05eaea843664dea6"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.417282 31890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:20.420353 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.421707 31890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:20.421867 31890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "48f32628127648b080afa57eae74ff64"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "ddec7ee632f5c9093b44b4440f253c97"
server_key_iv: "6c51d6a35f44c80a05eaea843664dea6"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.422024 31890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.454993 31890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.455849 31890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.456070 31890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.456743 31890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:20.457840 31890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:20.457912 31890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.457983 31890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:20.458033 31890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.467509 31890 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:37585
I20260810 14:36:20.467546 32030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:37585 every 8 connection(s)
I20260810 14:36:20.468557 31890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:20.477989 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.470577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:48769 (local address 127.21.43.126:37191)
0810 14:36:20.470733 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.470737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.471902 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 14:36:20.473311 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.473313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.473363 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.473439 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.474351 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.474905 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.476136 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.476356 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.476438 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.477521 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.477524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.477526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.477540 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.477559 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.477567 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.477633 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.477776 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.477831 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:36:20.478284 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 31890
I20260810 14:36:20.478397 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:20.478683 21677 external_mini_cluster.cc:1428] Setting key f7c654cc18dfe323116e9e6e250f16bd
I20260810 14:36:20.478979 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.470997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37191 (local address 127.21.43.65:48769)
0810 14:36:20.471768 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:20.471810 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:20.473103 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:20.473610 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:20.473617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:20.473701 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:20.474214 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.474237 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.475021 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.475023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.476019 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.476028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.476503 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.476506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.476650 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.477257 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:20.477258 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:20.477301 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:20.477368 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:20.477650 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:20.477657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:20.477661 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:36:20.477762 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:36:20.477994 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":195,"threads_started":1}
I20260810 14:36:20.480228 32031 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37191
I20260810 14:36:20.480491 32031 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:20.480731 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--builtin_ntp_servers=127.21.43.84:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:36:20.481103 32031 heartbeater.cc:511] Master 127.21.43.126:37191 requested a full tablet report, sending...
I20260810 14:36:20.482831 31819 ts_manager.cc:194] Registered new tserver with Master: 48f32628127648b080afa57eae74ff64 (127.21.43.65:37585)
I20260810 14:36:20.484114 31819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.65:48769
W20260810 14:36:20.600062 32034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.600456 32034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.600526 32034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.605082 32034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.605160 32034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.605221 32034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:36:20.610376 32034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:20.611714 32034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.613216 32034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.624075 32041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.624075 32040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.624385 32043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.624717 32034 server_base.cc:1034] running on GCE node
I20260810 14:36:20.625583 32034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.626370 32034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.627624 32034 hybrid_clock.cc:648] HybridClock initialized: now 1786372580627605 us; error 31 us; skew 500 ppm
I20260810 14:36:20.629961 32034 webserver.cc:460] Webserver started at http://127.21.43.66:35705/ using document root <none> and password file <none>
I20260810 14:36:20.630648 32034 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.630754 32034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.631019 32034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.633327 32034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "37b961a680894ec28d4d12b6a5cb46a8"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "a77ddc1b4c488ea565a9ed8562d974a9"
server_key_iv: "c54fe69fa6b3e93a556e6fe9bdad22df"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.633946 32034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "37b961a680894ec28d4d12b6a5cb46a8"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "a77ddc1b4c488ea565a9ed8562d974a9"
server_key_iv: "c54fe69fa6b3e93a556e6fe9bdad22df"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.638026 32034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:36:20.640924 32049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.642268 32034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:36:20.642413 32034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "37b961a680894ec28d4d12b6a5cb46a8"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "a77ddc1b4c488ea565a9ed8562d974a9"
server_key_iv: "c54fe69fa6b3e93a556e6fe9bdad22df"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.642542 32034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.665460 32034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.666263 32034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.666483 32034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.667204 32034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:20.668221 32034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:20.668298 32034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.668367 32034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:20.668417 32034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.677682 32034 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:43603
I20260810 14:36:20.677731 32176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:43603 every 8 connection(s)
I20260810 14:36:20.678671 32034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:20.679244 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32034
I20260810 14:36:20.679388 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:20.679754 21677 external_mini_cluster.cc:1428] Setting key 8d57f6316662a48f4f83c7af48f35e83
I20260810 14:36:20.681780 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--builtin_ntp_servers=127.21.43.84:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:36:20.688515 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.680680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:55133 (local address 127.21.43.126:37191)
0810 14:36:20.680842 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.680847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.682088 (+ 1241us) server_negotiation.cc:407] Connection header received
0810 14:36:20.683576 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.683580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.683641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.683750 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.684667 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.685208 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.686433 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.686660 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.686775 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.687985 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.687988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.687993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.688012 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.688032 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.688040 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.688128 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.688295 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.688362 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:36:20.689396 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.681193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37191 (local address 127.21.43.66:55133)
0810 14:36:20.681920 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:20.681975 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:20.683360 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:20.683932 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:20.683939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:20.684024 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:20.684513 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.684526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.685350 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.685358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.686298 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.686308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.686824 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.686828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.686999 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.687694 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:20.687696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:20.687740 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:20.687817 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:20.688124 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:20.688136 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:20.688144 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:36:20.688283 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 14:36:20.688572 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260810 14:36:20.690755 32177 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37191
I20260810 14:36:20.691045 32177 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:20.691813 32177 heartbeater.cc:511] Master 127.21.43.126:37191 requested a full tablet report, sending...
I20260810 14:36:20.693198 31819 ts_manager.cc:194] Registered new tserver with Master: 37b961a680894ec28d4d12b6a5cb46a8 (127.21.43.66:43603)
I20260810 14:36:20.693800 31819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.66:55133
W20260810 14:36:20.801481 32182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.801847 32182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.801924 32182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.806479 32182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.806574 32182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.806634 32182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:36:20.811930 32182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:37191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:20.813232 32182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.814795 32182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.826085 32189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.826054 32188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.826054 32191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.826372 32182 server_base.cc:1034] running on GCE node
I20260810 14:36:20.827181 32182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.827982 32182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.829164 32182 hybrid_clock.cc:648] HybridClock initialized: now 1786372580829130 us; error 55 us; skew 500 ppm
I20260810 14:36:20.831461 32182 webserver.cc:460] Webserver started at http://127.21.43.67:46307/ using document root <none> and password file <none>
I20260810 14:36:20.832103 32182 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.832170 32182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.832448 32182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.834750 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "006c12fa192547b587b2fdde64727b42"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "086702a0c1c654805ac438f39e6632f3"
server_key_iv: "dd9e8a71a22682cb0bcf8610886952a5"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.835435 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "006c12fa192547b587b2fdde64727b42"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "086702a0c1c654805ac438f39e6632f3"
server_key_iv: "dd9e8a71a22682cb0bcf8610886952a5"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.839650 32182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:36:20.842517 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.843959 32182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:20.844110 32182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "006c12fa192547b587b2fdde64727b42"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-chsk"
server_key: "086702a0c1c654805ac438f39e6632f3"
server_key_iv: "dd9e8a71a22682cb0bcf8610886952a5"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.844238 32182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.860389 32182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.861171 32182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.861387 32182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.862046 32182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:20.863061 32182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:20.863137 32182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.863260 32182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:20.863304 32182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.872444 32182 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:44615
I20260810 14:36:20.872465 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:44615 every 8 connection(s)
I20260810 14:36:20.873389 32182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:20.879834 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32182
I20260810 14:36:20.879963 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:20.880288 21677 external_mini_cluster.cc:1428] Setting key 224d288aebec7eaa70ee12d9b44c18d9
I20260810 14:36:20.883325 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.875285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:46945 (local address 127.21.43.126:37191)
0810 14:36:20.875446 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.875452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.876638 (+ 1186us) server_negotiation.cc:407] Connection header received
0810 14:36:20.878011 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.878014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.878066 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.878143 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.879097 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.879655 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.881102 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.881455 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.881606 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.882832 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.882835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.882837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.882849 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.882867 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.882875 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.882936 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.883098 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.883192 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:36:20.884235 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.875757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:37191 (local address 127.21.43.67:46945)
0810 14:36:20.876484 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:20.876533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:20.877811 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:20.878343 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:20.878350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:20.878433 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:20.878955 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.878969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.879785 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.879791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.880982 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.880993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.881665 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.881670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.881867 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.882565 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:20.882567 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:20.882626 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:20.882692 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:20.882954 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:20.882961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:20.882965 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:36:20.883080 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 14:36:20.883388 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260810 14:36:20.885491 32324 heartbeater.cc:347] Connected to a master server at 127.21.43.126:37191
I20260810 14:36:20.885761 32324 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:20.886325 32324 heartbeater.cc:511] Master 127.21.43.126:37191 requested a full tablet report, sending...
I20260810 14:36:20.887547 31819 ts_manager.cc:194] Registered new tserver with Master: 006c12fa192547b587b2fdde64727b42 (127.21.43.67:44615)
I20260810 14:36:20.888151 31819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.67:46945
I20260810 14:36:20.894407 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:36:20.897382 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:20.896484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.21.43.126:37191)
0810 14:36:20.896670 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.896674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.896728 (+ 54us) server_negotiation.cc:407] Connection header received
0810 14:36:20.896856 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.896858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.896900 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.896966 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.897251 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56582: BlockingRecv error: recv got EOF from 127.0.0.1:56582 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 14:36:20.897511 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31890
I20260810 14:36:20.908370 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32034
I20260810 14:36:20.918726 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32182
I20260810 14:36:20.928012 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 31786
2026-08-10T14:36:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (977 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T14:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:20Z Disabled control of system clock
I20260810 14:36:20.961938 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:42487
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:36:21.078070 32335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.078363 32335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.078402 32335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.082572 32335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:21.082648 32335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.082665 32335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.082682 32335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:21.082697 32335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:21.087709 32335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:42487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:21.088956 32335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.090454 32335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.099365 32341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.099397 32335 server_base.cc:1034] running on GCE node
W20260810 14:36:21.099360 32344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.099416 32342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.100241 32335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.101439 32335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.102631 32335 hybrid_clock.cc:648] HybridClock initialized: now 1786372581102612 us; error 32 us; skew 500 ppm
I20260810 14:36:21.104696 32335 webserver.cc:460] Webserver started at http://127.21.43.126:45543/ using document root <none> and password file <none>
I20260810 14:36:21.105302 32335 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.105401 32335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.105649 32335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.107898 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "50ce9861368c4efdb01f078972950e57"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "b61317d07bb4b10263c0116e5bed2421"
server_key_iv: "954ba2d7b8011bf9adea4d4f6a087e34"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.108498 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "50ce9861368c4efdb01f078972950e57"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "b61317d07bb4b10263c0116e5bed2421"
server_key_iv: "954ba2d7b8011bf9adea4d4f6a087e34"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.112269 32335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:36:21.114718 32351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.116035 32335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:36:21.116185 32335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "50ce9861368c4efdb01f078972950e57"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "b61317d07bb4b10263c0116e5bed2421"
server_key_iv: "954ba2d7b8011bf9adea4d4f6a087e34"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.116307 32335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.121191 32335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.121855 32335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.122067 32335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.129449 32335 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:42487
I20260810 14:36:21.129459 32407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:42487 every 8 connection(s)
I20260810 14:36:21.130518 32335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:21.133951 32408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:21.138336 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32335
I20260810 14:36:21.138445 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:36:21.138743 21677 external_mini_cluster.cc:1428] Setting key 9c393dfa519e9b2849ea3b4471c70e0b
I20260810 14:36:21.140144 32408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: Bootstrap starting.
I20260810 14:36:21.142601 32408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:21.143487 32408 log.cc:824] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:21.144665 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.140251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.21.43.126:42487)
0810 14:36:21.140921 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.140936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.140971 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:36:21.141627 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.141643 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.142004 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.142364 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.143188 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.143194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.143214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.143225 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.143279 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.143296 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.143419 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.143523 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.143782 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20260810 14:36:21.145938 32408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: No bootstrap required, opened a new log
I20260810 14:36:21.149775 32408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } }
I20260810 14:36:21.150094 32408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:21.150188 32408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ce9861368c4efdb01f078972950e57, State: Initialized, Role: FOLLOWER
I20260810 14:36:21.150655 32408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } }
I20260810 14:36:21.150826 32408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:21.150919 32408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:21.151026 32408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:21.152033 32408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } }
I20260810 14:36:21.152392 32408 leader_election.cc:302] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ce9861368c4efdb01f078972950e57; no voters:
I20260810 14:36:21.152724 32408 leader_election.cc:288] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:21.152841 32414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:21.153102 32414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [term 1 LEADER]: Becoming Leader. State: Replica: 50ce9861368c4efdb01f078972950e57, State: Running, Role: LEADER
I20260810 14:36:21.153452 32414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } }
I20260810 14:36:21.153960 32408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:21.155246 32416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ce9861368c4efdb01f078972950e57. Latest consensus state: current_term: 1 leader_uuid: "50ce9861368c4efdb01f078972950e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } } }
I20260810 14:36:21.155375 32416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:21.155508 32415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ce9861368c4efdb01f078972950e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ce9861368c4efdb01f078972950e57" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 42487 } } }
I20260810 14:36:21.155866 32415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57 [sys.catalog]: This master's current role is: LEADER
W20260810 14:36:21.158820 32430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:36:21.158892 32430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:36:21.158973 32425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:21.159791 32425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:21.165917 32425 catalog_manager.cc:1347] Generated new cluster ID: 1f616a05b8ce437c88ec93d7e0daa1e8
I20260810 14:36:21.165997 32425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:21.184057 32425 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:21.184471 32425 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:21.189960 32425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ce9861368c4efdb01f078972950e57: Generated new TSK 0
I20260810 14:36:21.190717 32425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:21.198508 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--builtin_ntp_servers=127.21.43.84:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:36:21.316833 32434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.317122 32434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.317165 32434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.321501 32434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.321570 32434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.321630 32434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:36:21.326678 32434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:21.327973 32434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.329419 32434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.339587 32440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.339589 32441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.339589 32443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.339668 32434 server_base.cc:1034] running on GCE node
I20260810 14:36:21.340344 32434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.341102 32434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.342325 32434 hybrid_clock.cc:648] HybridClock initialized: now 1786372581342309 us; error 31 us; skew 500 ppm
I20260810 14:36:21.344604 32434 webserver.cc:460] Webserver started at http://127.21.43.65:37857/ using document root <none> and password file <none>
I20260810 14:36:21.345363 32434 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.345475 32434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.345788 32434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.348376 32434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "852ddf681fc14719af20bce606bbb559"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "c1f500c6b5a11f82c5bc8ed89207f7c0"
server_key_iv: "d5920aae6ff1de5e59ee9365c08359d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.349045 32434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "852ddf681fc14719af20bce606bbb559"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "c1f500c6b5a11f82c5bc8ed89207f7c0"
server_key_iv: "d5920aae6ff1de5e59ee9365c08359d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.353379 32434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:21.356729 32450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.358059 32434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:36:21.358210 32434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "852ddf681fc14719af20bce606bbb559"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "c1f500c6b5a11f82c5bc8ed89207f7c0"
server_key_iv: "d5920aae6ff1de5e59ee9365c08359d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.358343 32434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.363586 32434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.364296 32434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.364503 32434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.365132 32434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:21.366143 32434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:21.366195 32434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.366237 32434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:21.366255 32434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.376164 32434 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:38127
I20260810 14:36:21.376197 32581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:38127 every 8 connection(s)
I20260810 14:36:21.377164 32434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:21.383131 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.379052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:34741 (local address 127.21.43.126:42487)
0810 14:36:21.379252 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.379257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.380303 (+ 1046us) server_negotiation.cc:407] Connection header received
0810 14:36:21.381534 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.381536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.381575 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.381656 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.382732 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.382735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.382737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.382747 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.382768 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.382777 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.382856 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.382977 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.383015 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:36:21.384059 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.379459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:42487 (local address 127.21.43.65:34741)
0810 14:36:21.380155 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.380195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.381353 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.381810 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.381816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.381911 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:21.382469 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:21.382471 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:21.382513 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:21.382579 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:21.382854 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:21.382861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:21.382865 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.383004 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.383306 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260810 14:36:21.385314 32582 heartbeater.cc:347] Connected to a master server at 127.21.43.126:42487
I20260810 14:36:21.385533 32582 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:21.385905 32582 heartbeater.cc:511] Master 127.21.43.126:42487 requested a full tablet report, sending...
I20260810 14:36:21.386483 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32434
I20260810 14:36:21.386620 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:21.386922 21677 external_mini_cluster.cc:1428] Setting key ebdf2aec9f8b35a8ef96a4f2b82dddea
I20260810 14:36:21.387709 32368 ts_manager.cc:194] Registered new tserver with Master: 852ddf681fc14719af20bce606bbb559 (127.21.43.65:38127)
I20260810 14:36:21.389158 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--builtin_ntp_servers=127.21.43.84:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:36:21.503460 32586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.503777 32586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.503856 32586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.508164 32586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.508245 32586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.508347 32586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:36:21.513198 32586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:21.514380 32586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.515905 32586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.525913 32593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.525913 32592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.525969 32595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.526218 32586 server_base.cc:1034] running on GCE node
I20260810 14:36:21.526662 32586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.527376 32586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.528554 32586 hybrid_clock.cc:648] HybridClock initialized: now 1786372581528537 us; error 32 us; skew 500 ppm
I20260810 14:36:21.530562 32586 webserver.cc:460] Webserver started at http://127.21.43.66:45473/ using document root <none> and password file <none>
I20260810 14:36:21.531231 32586 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.531302 32586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.531556 32586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.533668 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "af2a3012316046ceaa5fd1f044a2acfc"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "14b3cbceb3861abf2c68d5641ce4a8b9"
server_key_iv: "cc44a44668349a4f3cca3ab9759ce75e"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.534277 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "af2a3012316046ceaa5fd1f044a2acfc"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "14b3cbceb3861abf2c68d5641ce4a8b9"
server_key_iv: "cc44a44668349a4f3cca3ab9759ce75e"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.538347 32586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:36:21.541119 32602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.542438 32586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:36:21.542548 32586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "af2a3012316046ceaa5fd1f044a2acfc"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "14b3cbceb3861abf2c68d5641ce4a8b9"
server_key_iv: "cc44a44668349a4f3cca3ab9759ce75e"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.542707 32586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.547650 32586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.548277 32586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.548467 32586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.549024 32586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:21.549955 32586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:21.550004 32586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.550051 32586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:21.550109 32586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.559602 32725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34003 every 8 connection(s)
I20260810 14:36:21.559723 32586 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34003
I20260810 14:36:21.560683 32586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:21.566390 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32586
I20260810 14:36:21.566534 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:21.566893 21677 external_mini_cluster.cc:1428] Setting key 3e99e1e499ac30950642ff4e36ce8293
I20260810 14:36:21.567219 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.562608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:40167 (local address 127.21.43.126:42487)
0810 14:36:21.562775 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.562780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.563993 (+ 1213us) server_negotiation.cc:407] Connection header received
0810 14:36:21.565382 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.565385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.565430 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.565517 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.566673 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.566676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.566678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.566690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.566716 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.566725 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.566812 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.567003 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.567058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:36:21.568252 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.563105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:42487 (local address 127.21.43.66:40167)
0810 14:36:21.563841 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.563882 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.565109 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.565650 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.565657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.565742 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:21.566324 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:21.566327 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:21.566385 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:21.566475 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:21.566820 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:21.566830 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:21.566837 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.567028 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.567308 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
I20260810 14:36:21.569137 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--builtin_ntp_servers=127.21.43.84:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 14:36:21.569509 32726 heartbeater.cc:347] Connected to a master server at 127.21.43.126:42487
I20260810 14:36:21.569765 32726 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:21.570182 32726 heartbeater.cc:511] Master 127.21.43.126:42487 requested a full tablet report, sending...
I20260810 14:36:21.571383 32368 ts_manager.cc:194] Registered new tserver with Master: af2a3012316046ceaa5fd1f044a2acfc (127.21.43.66:34003)
W20260810 14:36:21.688200 32730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.688493 32730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.688536 32730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.692780 32730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.692847 32730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.692939 32730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:36:21.697890 32730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:42487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:21.699030 32730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.700556 32730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.710615 32736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.710644 32739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.710781 32730 server_base.cc:1034] running on GCE node
W20260810 14:36:21.710673 32735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.711345 32730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.712023 32730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.713222 32730 hybrid_clock.cc:648] HybridClock initialized: now 1786372581713208 us; error 26 us; skew 500 ppm
I20260810 14:36:21.715097 32730 webserver.cc:460] Webserver started at http://127.21.43.67:36137/ using document root <none> and password file <none>
I20260810 14:36:21.715745 32730 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.715847 32730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.716066 32730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.718230 32730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "b2e5dddc6464498e8ae699808f6be1ba"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "9dddc9077960d9a3bf6a6dd5c3b13534"
server_key_iv: "42b3372b6e07108ecc5395476546b5a7"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.718811 32730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e5dddc6464498e8ae699808f6be1ba"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "9dddc9077960d9a3bf6a6dd5c3b13534"
server_key_iv: "42b3372b6e07108ecc5395476546b5a7"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.722860 32730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 14:36:21.725667 32746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.726970 32730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:36:21.727169 32730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "b2e5dddc6464498e8ae699808f6be1ba"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-chsk"
server_key: "9dddc9077960d9a3bf6a6dd5c3b13534"
server_key_iv: "42b3372b6e07108ecc5395476546b5a7"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.727312 32730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.734448 32730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.735270 32730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.735463 32730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.736161 32730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:21.737186 32730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:21.737243 32730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.737283 32730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:21.737298 32730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.746445 32730 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:33743
I20260810 14:36:21.746516 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:33743 every 8 connection(s)
I20260810 14:36:21.747617 32730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:21.753649 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.749471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:38327 (local address 127.21.43.126:42487)
0810 14:36:21.749646 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.749651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.750700 (+ 1049us) server_negotiation.cc:407] Connection header received
0810 14:36:21.752053 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.752056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.752095 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.752172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.753254 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.753257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.753259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.753271 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.753292 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.753301 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.753382 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.753500 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.753539 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:36:21.754617 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.749886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:42487 (local address 127.21.43.67:38327)
0810 14:36:21.750550 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.750589 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.751870 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.752332 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.752338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.752421 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:21.752962 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:21.752963 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:21.753003 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:21.753089 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:21.753381 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:21.753387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:21.753391 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.753568 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.753844 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260810 14:36:21.755980 405 heartbeater.cc:347] Connected to a master server at 127.21.43.126:42487
I20260810 14:36:21.756254 405 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:21.756629 405 heartbeater.cc:511] Master 127.21.43.126:42487 requested a full tablet report, sending...
I20260810 14:36:21.757664 32368 ts_manager.cc:194] Registered new tserver with Master: b2e5dddc6464498e8ae699808f6be1ba (127.21.43.67:33743)
I20260810 14:36:21.757913 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 32730
I20260810 14:36:21.758025 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:21.758312 21677 external_mini_cluster.cc:1428] Setting key b7f7e32d534af389954047ffe99b1f1e
I20260810 14:36:21.760354 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:36:21.763489 32411 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:36:21.762472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.21.43.126:42487)
0810 14:36:21.762632 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.762637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.762774 (+ 137us) server_negotiation.cc:407] Connection header received
0810 14:36:21.762926 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.762929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.762972 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.763060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.763314 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52074: BlockingRecv error: recv got EOF from 127.0.0.1:52074 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:36:21.763890 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32434
I20260810 14:36:21.773451 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32586
I20260810 14:36:21.783284 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32730
I20260810 14:36:21.793524 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 32335
2026-08-10T14:36:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (865 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:21 dist-test-slave-chsk krb5kdc[417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:21 dist-test-slave-chsk krb5kdc[417](info): set up 2 sockets
Aug 10 14:36:21 dist-test-slave-chsk krb5kdc[417](info): commencing operation
krb5kdc: starting...
W20260810 14:36:22.912616 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:36:23.046144 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43757
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:23.160427 433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:23.160755 433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:23.160835 433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:23.164872 433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:23.164971 433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:23.165010 433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:23.165071 433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:23.165112 433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:23.170490 433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:23.171686 433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:23.172749 433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:23.181339 439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.181375 440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.181339 442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:23.181675 433 server_base.cc:1034] running on GCE node
I20260810 14:36:23.182401 433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:23.183630 433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:23.184793 433 hybrid_clock.cc:648] HybridClock initialized: now 1786372583184764 us; error 43 us; skew 500 ppm
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:23.192888 433 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:36:23.193846 433 webserver.cc:460] Webserver started at http://127.21.43.126:37943/ using document root <none> and password file <none>
I20260810 14:36:23.194440 433 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:23.194515 433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:23.194747 433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:23.196897 433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "a08343d68d504b16b69cc0e31ab1b305"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "7b57df5fb9e2b395f4e633090fa331c4"
server_key_iv: "aac50a6a36656fe1211566f4ef3ff3d4"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.197482 433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "a08343d68d504b16b69cc0e31ab1b305"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "7b57df5fb9e2b395f4e633090fa331c4"
server_key_iv: "aac50a6a36656fe1211566f4ef3ff3d4"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.201272 433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 14:36:23.203774 450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.205015 433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:23.205178 433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "a08343d68d504b16b69cc0e31ab1b305"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "7b57df5fb9e2b395f4e633090fa331c4"
server_key_iv: "aac50a6a36656fe1211566f4ef3ff3d4"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.205300 433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:23.224634 433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:23.228003 433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:23.228235 433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:23.235934 433 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43757
I20260810 14:36:23.235960 508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43757 every 8 connection(s)
I20260810 14:36:23.236910 433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:23.239758 509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:23.243780 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 433
I20260810 14:36:23.243917 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:36:23.244215 21677 external_mini_cluster.cc:1428] Setting key 517df57593c899bfdecc192325891bee
I20260810 14:36:23.245854 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: Bootstrap starting.
I20260810 14:36:23.248397 509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:23.249151 509 log.cc:824] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:23.251317 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: No bootstrap required, opened a new log
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:23.255224 509 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } }
I20260810 14:36:23.255478 509 raft_consensus.cc:374] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:23.255574 509 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08343d68d504b16b69cc0e31ab1b305, State: Initialized, Role: FOLLOWER
I20260810 14:36:23.256138 509 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } }
I20260810 14:36:23.256325 509 raft_consensus.cc:388] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:23.256407 509 raft_consensus.cc:482] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:23.256510 509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:23.257766 509 raft_consensus.cc:504] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } }
I20260810 14:36:23.258134 509 leader_election.cc:302] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08343d68d504b16b69cc0e31ab1b305; no voters:
I20260810 14:36:23.258476 509 leader_election.cc:288] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:23.258587 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.245587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.21.43.126:43757)
0810 14:36:23.245988 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:23.246011 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:23.246061 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:36:23.246676 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:23.246692 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:23.246997 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:23.247289 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:23.248119 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.249190 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.250050 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.250643 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.250832 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.253406 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:23.253412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:23.253425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:23.253452 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:23.255272 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.255801 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.255807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.255813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.255964 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.256247 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.256250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.256252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.256811 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:23.257017 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:23.257276 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 14:36:23.257594 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
I20260810 14:36:23.258631 514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:23.258879 514 raft_consensus.cc:686] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [term 1 LEADER]: Becoming Leader. State: Replica: a08343d68d504b16b69cc0e31ab1b305, State: Running, Role: LEADER
I20260810 14:36:23.259228 514 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } }
I20260810 14:36:23.259826 509 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:23.260413 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08343d68d504b16b69cc0e31ab1b305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } } }
I20260810 14:36:23.260524 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:23.260771 516 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08343d68d504b16b69cc0e31ab1b305. Latest consensus state: current_term: 1 leader_uuid: "a08343d68d504b16b69cc0e31ab1b305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08343d68d504b16b69cc0e31ab1b305" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43757 } } }
I20260810 14:36:23.261426 516 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:23.261699 520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:23.264639 520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:23.269909 520 catalog_manager.cc:1347] Generated new cluster ID: 444ae67e1746433491bda1baf181ad40
I20260810 14:36:23.269982 520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:23.292515 520 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:23.293473 520 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:23.298345 520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: Generated new TSK 0
I20260810 14:36:23.299043 520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:36:23.335247 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:23.451326 536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:23.451627 536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:23.451707 536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:23.455714 536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:23.455804 536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:23.455904 536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:36:23.461314 536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:23.462459 536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:23.463582 536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:23.472998 544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.472998 541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.473054 542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:23.473212 536 server_base.cc:1034] running on GCE node
I20260810 14:36:23.473713 536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:23.474342 536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:23.475504 536 hybrid_clock.cc:648] HybridClock initialized: now 1786372583475470 us; error 51 us; skew 500 ppm
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:23.478478 536 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:36:23.479490 536 webserver.cc:460] Webserver started at http://127.21.43.65:42411/ using document root <none> and password file <none>
I20260810 14:36:23.480058 536 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:23.480151 536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:23.480388 536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:23.482415 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "a7fc2b9fa9fc4506827a1b84b50284dc"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "1ca7f2a08b0b21745dbefb93b2aa9bf8"
server_key_iv: "4248d9c24035c163c120aa2fb0908265"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.482977 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "a7fc2b9fa9fc4506827a1b84b50284dc"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "1ca7f2a08b0b21745dbefb93b2aa9bf8"
server_key_iv: "4248d9c24035c163c120aa2fb0908265"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.486621 536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:36:23.489032 553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.490212 536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:23.490346 536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "a7fc2b9fa9fc4506827a1b84b50284dc"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "1ca7f2a08b0b21745dbefb93b2aa9bf8"
server_key_iv: "4248d9c24035c163c120aa2fb0908265"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.490454 536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:23.501916 536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:23.504706 536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:23.504899 536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:23.505471 536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:23.506258 536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:23.506328 536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.506402 536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:23.506444 536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.515548 536 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:46117
I20260810 14:36:23.515579 680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:46117 every 8 connection(s)
I20260810 14:36:23.516434 536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:23.523023 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 536
I20260810 14:36:23.523180 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:23.523468 21677 external_mini_cluster.cc:1428] Setting key 368dd88aa1210b5e7794d1b99880b1d2
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:23.530877 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.518317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:60987 (local address 127.21.43.126:43757)
0810 14:36:23.518499 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:23.518503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:23.519347 (+ 844us) server_negotiation.cc:407] Connection header received
0810 14:36:23.520789 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:23.520793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:23.520856 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:23.520970 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:23.522179 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.522737 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.524096 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.524344 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.524462 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.526945 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:23.526949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:23.526951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:23.526975 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:23.528484 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.529020 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.529024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.529026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.529115 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.529503 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.529510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.529513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.529800 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:23.529918 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:23.530480 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 14:36:23.530637 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:36:23.531627 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.518698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43757 (local address 127.21.43.65:60987)
0810 14:36:23.519197 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:23.519249 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:23.520571 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:23.521163 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:23.521170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:23.521534 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:23.522020 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.522040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.522882 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.522885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.523933 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.523944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.524504 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.524509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.524680 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.525301 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:23.525317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:23.526770 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:23.528622 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.528627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.528642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.528910 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.529213 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.529218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.529220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.529363 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.529949 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:23.529955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:23.530206 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 14:36:23.530383 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 14:36:23.530651 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260810 14:36:23.533004 683 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43757
I20260810 14:36:23.533272 683 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:36:23.533833 683 heartbeater.cc:511] Master 127.21.43.126:43757 requested a full tablet report, sending...
I20260810 14:36:23.535465 469 ts_manager.cc:194] Registered new tserver with Master: a7fc2b9fa9fc4506827a1b84b50284dc (127.21.43.65:46117)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:36:23.536695 469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:60987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:36:23.554150 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:23.669226 688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:23.669536 688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:23.669617 688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:23.673718 688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:23.673810 688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:23.674064 688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:36:23.679785 688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:23.680936 688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:23.682618 688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:23.692607 696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.692623 693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:23.692667 688 server_base.cc:1034] running on GCE node
W20260810 14:36:23.692626 694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:23.693284 688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:23.693940 688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:23.695199 688 hybrid_clock.cc:648] HybridClock initialized: now 1786372583695164 us; error 45 us; skew 500 ppm
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:23.697918 688 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:36:23.698858 688 webserver.cc:460] Webserver started at http://127.21.43.66:40899/ using document root <none> and password file <none>
I20260810 14:36:23.699496 688 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:23.699589 688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:23.699824 688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:23.701876 688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "f978639c5772dd3695cf192feac2ddec"
server_key_iv: "8b378c91b41fe65fdecb88e104689ae5"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.702478 688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "f978639c5772dd3695cf192feac2ddec"
server_key_iv: "8b378c91b41fe65fdecb88e104689ae5"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.706300 688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:36:23.708848 703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.710202 688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:36:23.710338 688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "f978639c5772dd3695cf192feac2ddec"
server_key_iv: "8b378c91b41fe65fdecb88e104689ae5"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.710445 688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:23.721570 688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:23.724555 688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:23.724761 688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:23.725379 688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:23.726227 688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:23.726275 688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.726343 688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:23.726377 688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.738196 688 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:37877
I20260810 14:36:23.738335 832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:37877 every 8 connection(s)
I20260810 14:36:23.739384 688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:23.741431 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 688
I20260810 14:36:23.741567 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:23.741890 21677 external_mini_cluster.cc:1428] Setting key d35249b67d58f71cbfe53305c0e8f7c6
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:36:23.754944 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.741252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:41877 (local address 127.21.43.126:43757)
0810 14:36:23.741397 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:23.741401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:23.742419 (+ 1018us) server_negotiation.cc:407] Connection header received
0810 14:36:23.744022 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:23.744028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:23.744092 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:23.744187 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:23.745352 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.746019 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.747598 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.748023 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.748153 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.750831 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:23.750841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:23.750843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:23.750869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:23.752603 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.753129 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.753133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.753135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.753238 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.753602 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.753605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.753607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.753808 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:23.753918 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:23.754542 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 14:36:23.754683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:36:23.755973 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.741730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43757 (local address 127.21.43.66:41877)
0810 14:36:23.742252 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:23.742295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:23.743803 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:23.744339 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:23.744346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:23.744723 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:23.745192 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.745205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.746154 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.746169 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.747468 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.747478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.748179 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.748181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.748325 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.748973 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:23.748990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:23.750653 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:23.752725 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.752731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.752747 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.753020 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.753337 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.753341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.753343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.753490 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.753930 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:23.753939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:23.754294 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 14:36:23.754497 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 14:36:23.754869 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":160,"threads_started":1}
I20260810 14:36:23.757228 833 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43757
I20260810 14:36:23.757480 833 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:23.758148 833 heartbeater.cc:511] Master 127.21.43.126:43757 requested a full tablet report, sending...
I20260810 14:36:23.759465 469 ts_manager.cc:194] Registered new tserver with Master: b1e0af3a4ba749b891dc7b2daa0f8f5e (127.21.43.66:37877)
I20260810 14:36:23.760093 469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:41877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:36:23.774173 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:23.892081 839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:23.892372 839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:23.892418 839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:23.896687 839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:23.896754 839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:23.896850 839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:36:23.902410 839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:36:23.903630 839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:23.904701 839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:23.914812 847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.914822 845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:23.914788 844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:23.915172 839 server_base.cc:1034] running on GCE node
I20260810 14:36:23.915683 839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:23.916317 839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:23.917527 839 hybrid_clock.cc:648] HybridClock initialized: now 1786372583917494 us; error 51 us; skew 500 ppm
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:23.920524 839 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:36:23.921475 839 webserver.cc:460] Webserver started at http://127.21.43.67:35409/ using document root <none> and password file <none>
I20260810 14:36:23.922032 839 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:23.922097 839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:23.922283 839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:23.924412 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "0637b06d8e3c463cb5b1e5daad43f501"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "9ebd4fabf73de31aea179a03f8145981"
server_key_iv: "e88ad73ff9bae66149b406073d3badce"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.924968 839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "0637b06d8e3c463cb5b1e5daad43f501"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "9ebd4fabf73de31aea179a03f8145981"
server_key_iv: "e88ad73ff9bae66149b406073d3badce"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.928965 839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:23.931641 855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.932905 839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:23.933053 839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "0637b06d8e3c463cb5b1e5daad43f501"
format_stamp: "Formatted at 2026-08-10 14:36:23 on dist-test-slave-chsk"
server_key: "9ebd4fabf73de31aea179a03f8145981"
server_key_iv: "e88ad73ff9bae66149b406073d3badce"
server_key_version: "encryptionkey@0"
I20260810 14:36:23.933185 839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:23.947857 839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:23.951057 839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:23.951324 839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:23.952095 839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:23.953052 839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:23.953101 839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.953176 839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:23.953209 839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:23.962934 839 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:37935
I20260810 14:36:23.962955 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:37935 every 8 connection(s)
I20260810 14:36:23.964006 839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:23.972048 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 839
I20260810 14:36:23.972239 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:23.972545 21677 external_mini_cluster.cc:1428] Setting key b4976581dd17c930c03db029d23e73ab
Aug 10 14:36:23 dist-test-slave-chsk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:36:23.978477 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.966032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:47947 (local address 127.21.43.126:43757)
0810 14:36:23.966180 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:23.966184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:23.967022 (+ 838us) server_negotiation.cc:407] Connection header received
0810 14:36:23.968621 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:23.968624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:23.968699 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:23.968797 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:23.970097 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.970658 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.971931 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.972150 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.972260 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.975181 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:23.975185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:23.975188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:23.975218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:23.976517 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.977049 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.977051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.977053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.977129 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.977484 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.977486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.977488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.977674 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:23.977752 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:23.978239 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 14:36:23.978348 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:23.979473 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.966338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43757 (local address 127.21.43.67:47947)
0810 14:36:23.966849 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:23.966896 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:23.968377 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:23.968993 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:23.969000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:23.969417 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:23.969935 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.969949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.970796 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.970800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.971783 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:23.971794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.972325 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.972330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:23.972505 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.973237 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:23.973254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:23.974987 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:23.976639 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.976644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.976658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.976953 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.977233 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:23.977236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:23.977238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:23.977396 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:23.977763 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:23.977768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:23.978027 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 14:36:23.978194 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 14:36:23.978469 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260810 14:36:23.980590 984 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43757
I20260810 14:36:23.980883 984 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:23.981405 984 heartbeater.cc:511] Master 127.21.43.126:43757 requested a full tablet report, sending...
I20260810 14:36:23.982585 469 ts_manager.cc:194] Registered new tserver with Master: 0637b06d8e3c463cb5b1e5daad43f501 (127.21.43.67:37935)
I20260810 14:36:23.983278 469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:47947
I20260810 14:36:23.986971 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:23.997339 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:23.989185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.21.43.126:43757)
0810 14:36:23.989319 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:23.989323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:23.989446 (+ 123us) server_negotiation.cc:407] Connection header received
0810 14:36:23.989552 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:23.989555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:23.989596 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:23.989675 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:23.990387 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.990913 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.991849 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:23.992069 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:23.992173 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:23.993628 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:23.993637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:23.993639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:23.993664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:23.995321 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.995893 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.995913 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.995916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.996031 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:23.996369 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:23.996372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:23.996374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:23.996636 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:23.996786 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:23.997024 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 14:36:23.997160 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:36:24.000504 469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:24.003738 469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:24.019429 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.013128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.21.43.65:46117)
0810 14:36:24.013495 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.013516 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.013899 (+ 383us) server_negotiation.cc:407] Connection header received
0810 14:36:24.014594 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.014600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.014843 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.014983 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:24.015461 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016249 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.017798 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.018665 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.018792 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.018950 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.019132 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.019255 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260810 14:36:24.019424 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.012919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:46117 (local address 127.0.0.1:37884)
0810 14:36:24.013744 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.013800 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:24.014457 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:24.015189 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:24.015192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:24.015213 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:24.015327 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.015333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016384 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.016389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.017685 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.017696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.018794 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.018798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.018924 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.018968 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 14:36:24.019086 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:36:24.019246 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260810 14:36:24.019497 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.013138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:37877 (local address 127.0.0.1:33166)
0810 14:36:24.013831 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.013847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:24.013974 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:24.015104 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:24.015112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:24.015174 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:24.015317 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.015323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016815 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.016818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.017909 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.017916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.019015 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.019019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.019124 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.019187 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 14:36:24.019263 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:36:24.019343 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":4,"spinlock_wait_cycles":2816,"thread_start_us":152,"threads_started":1}
I20260810 14:36:24.019981 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.013567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:37935 (local address 127.0.0.1:47932)
0810 14:36:24.014085 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.014099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:24.014207 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:24.015313 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:24.015315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:24.015329 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:24.015426 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.015432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016853 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.016856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.018335 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.018347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.019599 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.019604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.019698 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.019720 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:36:24.019790 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:36:24.019858 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":162,"thread_start_us":118,"threads_started":1}
I20260810 14:36:24.020049 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.013716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.21.43.66:37877)
0810 14:36:24.014317 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.014328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.014355 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:36:24.014442 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.014447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.014674 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.014836 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:24.015437 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016658 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.018019 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.018700 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.018803 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.018951 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.019795 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.019891 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":113,"threads_started":1}
I20260810 14:36:24.020516 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.014024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.21.43.67:37935)
0810 14:36:24.014714 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.014726 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.014750 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:36:24.014823 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.014827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.014984 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.015128 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:24.015549 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.016683 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.018474 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.019296 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.019387 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.019516 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.020331 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.020406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":196,"threads_started":1}
I20260810 14:36:24.022900 735 tablet_service.cc:1467] Processing CreateTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b (DEFAULT_TABLE table=test-table [id=4375bdc48ff4444984a0e759078d1824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:24.023038 905 tablet_service.cc:1467] Processing CreateTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b (DEFAULT_TABLE table=test-table [id=4375bdc48ff4444984a0e759078d1824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:24.024008 735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11784af0e54e4f9d93db40f98f6f7a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:24.024015 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11784af0e54e4f9d93db40f98f6f7a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:24.030200 1002 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Bootstrap starting.
I20260810 14:36:24.031524 1003 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Bootstrap starting.
I20260810 14:36:24.032922 606 tablet_service.cc:1467] Processing CreateTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b (DEFAULT_TABLE table=test-table [id=4375bdc48ff4444984a0e759078d1824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:24.033727 1003 tablet_bootstrap.cc:654] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:24.033727 1002 tablet_bootstrap.cc:654] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:24.034204 606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11784af0e54e4f9d93db40f98f6f7a0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:24.034574 1003 log.cc:824] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:24.034611 1002 log.cc:824] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:24.036914 1003 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: No bootstrap required, opened a new log
I20260810 14:36:24.037148 1003 ts_tablet_manager.cc:1397] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:36:24.040563 1002 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: No bootstrap required, opened a new log
I20260810 14:36:24.040856 1002 ts_tablet_manager.cc:1397] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 14:36:24.041317 1003 raft_consensus.cc:348] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.041610 1003 raft_consensus.cc:374] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:24.041718 1003 raft_consensus.cc:729] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0637b06d8e3c463cb5b1e5daad43f501, State: Initialized, Role: FOLLOWER
I20260810 14:36:24.042260 1003 consensus_queue.cc:260] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.042747 1007 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Bootstrap starting.
I20260810 14:36:24.043475 1003 ts_tablet_manager.cc:1428] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260810 14:36:24.043936 984 heartbeater.cc:503] Master 127.21.43.126:43757 was elected leader, sending a full tablet report...
I20260810 14:36:24.045171 1007 tablet_bootstrap.cc:654] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:24.045007 1002 raft_consensus.cc:348] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.045288 1002 raft_consensus.cc:374] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:24.045367 1002 raft_consensus.cc:729] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e0af3a4ba749b891dc7b2daa0f8f5e, State: Initialized, Role: FOLLOWER
I20260810 14:36:24.045804 1002 consensus_queue.cc:260] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.046085 1007 log.cc:824] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:24.046437 833 heartbeater.cc:503] Master 127.21.43.126:43757 was elected leader, sending a full tablet report...
I20260810 14:36:24.046775 1002 ts_tablet_manager.cc:1428] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:36:24.048409 1007 tablet_bootstrap.cc:492] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: No bootstrap required, opened a new log
I20260810 14:36:24.048636 1007 ts_tablet_manager.cc:1397] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:36:24.052809 1007 raft_consensus.cc:348] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.053058 1007 raft_consensus.cc:374] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:24.053117 1007 raft_consensus.cc:729] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fc2b9fa9fc4506827a1b84b50284dc, State: Initialized, Role: FOLLOWER
I20260810 14:36:24.053510 1007 consensus_queue.cc:260] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.054154 683 heartbeater.cc:503] Master 127.21.43.126:43757 was elected leader, sending a full tablet report...
I20260810 14:36:24.054399 1007 ts_tablet_manager.cc:1428] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 14:36:24.215183 985 tablet.cc:2369] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:24.240659 834 tablet.cc:2369] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:24.267860 684 tablet.cc:2369] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:24.317595 1011 raft_consensus.cc:482] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:24.317804 1011 raft_consensus.cc:504] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.318905 1011 leader_election.cc:288] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e0af3a4ba749b891dc7b2daa0f8f5e (127.21.43.66:37877), 0637b06d8e3c463cb5b1e5daad43f501 (127.21.43.67:37935)
I20260810 14:36:24.323024 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.319335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:45673 (local address 127.21.43.66:37877)
0810 14:36:24.319471 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.319476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.319759 (+ 283us) server_negotiation.cc:407] Connection header received
0810 14:36:24.319920 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.319923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.319977 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.320063 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:24.320328 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.320839 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322000 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.322404 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322529 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.322595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.322823 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.322894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:24.323024 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.319350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:35687 (local address 127.21.43.67:37935)
0810 14:36:24.319471 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.319476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.319852 (+ 376us) server_negotiation.cc:407] Connection header received
0810 14:36:24.320008 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.320011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.320067 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.320150 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:24.320412 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.320900 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322021 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.322420 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322552 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.322608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.322823 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.322893 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:36:24.323052 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.319261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:37877 (local address 127.21.43.65:45673)
0810 14:36:24.319650 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.319663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:24.319813 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:24.320083 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:24.320086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:24.320102 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:24.320221 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.320226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.320953 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.320956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.321871 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.321880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.322528 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.322643 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.322723 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 14:36:24.322806 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:36:24.322890 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":115,"thread_start_us":130,"threads_started":1}
I20260810 14:36:24.323061 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.319261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:37935 (local address 127.21.43.65:35687)
0810 14:36:24.319752 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.319765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:24.319887 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:24.320163 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:24.320165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:24.320180 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:24.320306 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.320311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.321026 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.321029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.321860 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:24.321868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.322541 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.322544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:24.322674 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.322721 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:36:24.322805 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:36:24.322888 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":56,"thread_start_us":99,"threads_started":1}
I20260810 14:36:24.323778 927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11784af0e54e4f9d93db40f98f6f7a0b" candidate_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0637b06d8e3c463cb5b1e5daad43f501" is_pre_election: true
I20260810 14:36:24.323832 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11784af0e54e4f9d93db40f98f6f7a0b" candidate_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" is_pre_election: true
I20260810 14:36:24.324103 927 raft_consensus.cc:2454] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7fc2b9fa9fc4506827a1b84b50284dc in term 0.
I20260810 14:36:24.324110 780 raft_consensus.cc:2454] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7fc2b9fa9fc4506827a1b84b50284dc in term 0.
I20260810 14:36:24.324540 555 leader_election.cc:302] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0637b06d8e3c463cb5b1e5daad43f501, a7fc2b9fa9fc4506827a1b84b50284dc; no voters:
I20260810 14:36:24.324834 1011 raft_consensus.cc:2781] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:24.324923 1011 raft_consensus.cc:482] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:24.324968 1011 raft_consensus.cc:3037] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:24.326077 1011 raft_consensus.cc:504] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.326402 1011 leader_election.cc:288] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [CANDIDATE]: Term 1 election: Requested vote from peers b1e0af3a4ba749b891dc7b2daa0f8f5e (127.21.43.66:37877), 0637b06d8e3c463cb5b1e5daad43f501 (127.21.43.67:37935)
I20260810 14:36:24.326745 927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11784af0e54e4f9d93db40f98f6f7a0b" candidate_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0637b06d8e3c463cb5b1e5daad43f501"
I20260810 14:36:24.326745 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11784af0e54e4f9d93db40f98f6f7a0b" candidate_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e"
I20260810 14:36:24.326879 780 raft_consensus.cc:3037] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:24.326874 927 raft_consensus.cc:3037] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:24.328202 927 raft_consensus.cc:2454] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7fc2b9fa9fc4506827a1b84b50284dc in term 1.
I20260810 14:36:24.328202 780 raft_consensus.cc:2454] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7fc2b9fa9fc4506827a1b84b50284dc in term 1.
I20260810 14:36:24.328517 555 leader_election.cc:302] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0637b06d8e3c463cb5b1e5daad43f501, a7fc2b9fa9fc4506827a1b84b50284dc; no voters:
I20260810 14:36:24.328691 1011 raft_consensus.cc:2781] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:24.328894 1011 raft_consensus.cc:686] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [term 1 LEADER]: Becoming Leader. State: Replica: a7fc2b9fa9fc4506827a1b84b50284dc, State: Running, Role: LEADER
I20260810 14:36:24.329214 1011 consensus_queue.cc:237] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } }
I20260810 14:36:24.332298 468 catalog_manager.cc:5495] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc reported cstate change: term changed from 0 to 1, leader changed from <none> to a7fc2b9fa9fc4506827a1b84b50284dc (127.21.43.65). New cstate: current_term: 1 leader_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7fc2b9fa9fc4506827a1b84b50284dc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 46117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 } health_report { overall_health: UNKNOWN } } }
I20260810 14:36:24.383805 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.380406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.21.43.65:46117)
0810 14:36:24.380524 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.380528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.380540 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:36:24.380617 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.380620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.380675 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.380764 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:24.380975 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.381470 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.382355 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.382578 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.382663 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.383002 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:24.383485 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:24.383577 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.383646 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.383696 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:36:24.389765 780 raft_consensus.cc:1261] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 1 FOLLOWER]: Refusing update from remote peer a7fc2b9fa9fc4506827a1b84b50284dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:24.389765 927 raft_consensus.cc:1261] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 1 FOLLOWER]: Refusing update from remote peer a7fc2b9fa9fc4506827a1b84b50284dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:24.390520 1015 consensus_queue.cc:1035] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e0af3a4ba749b891dc7b2daa0f8f5e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:24.390758 1011 consensus_queue.cc:1035] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0637b06d8e3c463cb5b1e5daad43f501" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:24.398269 1021 mvcc.cc:205] Tried to move back new op lower bound from 7316982105653075968 to 7316982105414955008. Current Snapshot: MvccSnapshot[applied={T|T < 7316982105653075968}]
I20260810 14:36:24.398528 1022 mvcc.cc:205] Tried to move back new op lower bound from 7316982105653075968 to 7316982105414955008. Current Snapshot: MvccSnapshot[applied={T|T < 7316982105653075968}]
I20260810 14:36:24.401576 1023 mvcc.cc:205] Tried to move back new op lower bound from 7316982105653075968 to 7316982105414955008. Current Snapshot: MvccSnapshot[applied={T|T < 7316982105653075968 or (T in {7316982105653075968})}]
I20260810 14:36:24.410988 469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:24.411249 469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:24.414047 469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a08343d68d504b16b69cc0e31ab1b305: Sending DeleteTablet for 3 replicas of tablet 11784af0e54e4f9d93db40f98f6f7a0b
I20260810 14:36:24.414875 606 tablet_service.cc:1514] Processing DeleteTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:24 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:37884
I20260810 14:36:24.414880 735 tablet_service.cc:1514] Processing DeleteTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:24 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:33166
I20260810 14:36:24.414995 905 tablet_service.cc:1514] Processing DeleteTablet for tablet 11784af0e54e4f9d93db40f98f6f7a0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:24 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:47932
I20260810 14:36:24.415521 1032 tablet_replica.cc:330] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: stopping tablet replica
I20260810 14:36:24.415683 1034 tablet_replica.cc:330] T 11784af0e54e4f9d93db40f98f6f7a0b P a7fc2b9fa9fc4506827a1b84b50284dc: stopping tablet replica
I20260810 14:36:24.415937 1032 raft_consensus.cc:2229] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:24.416198 1033 tablet_replica.cc:330] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: stopping tablet replica
I20260810 14:36:24.416378 1032 raft_consensus.cc:2258] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:24.416556 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 536
I20260810 14:36:24.416566 1033 raft_consensus.cc:2229] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:24.417006 1033 raft_consensus.cc:2258] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:24.419258 1032 ts_tablet_manager.cc:1905] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:24.421010 1033 ts_tablet_manager.cc:1905] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:24.423353 1032 ts_tablet_manager.cc:1918] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:24.423569 1032 log.cc:1196] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/11784af0e54e4f9d93db40f98f6f7a0b
I20260810 14:36:24.423933 1032 ts_tablet_manager.cc:1939] T 11784af0e54e4f9d93db40f98f6f7a0b P b1e0af3a4ba749b891dc7b2daa0f8f5e: Deleting consensus metadata
I20260810 14:36:24.424527 1033 ts_tablet_manager.cc:1918] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:24.424645 1033 log.cc:1196] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/11784af0e54e4f9d93db40f98f6f7a0b
I20260810 14:36:24.425027 1033 ts_tablet_manager.cc:1939] T 11784af0e54e4f9d93db40f98f6f7a0b P 0637b06d8e3c463cb5b1e5daad43f501: Deleting consensus metadata
I20260810 14:36:24.425253 453 catalog_manager.cc:4841] TS b1e0af3a4ba749b891dc7b2daa0f8f5e (127.21.43.66:37877): tablet 11784af0e54e4f9d93db40f98f6f7a0b (table test-table [id=4375bdc48ff4444984a0e759078d1824]) successfully deleted
I20260810 14:36:24.426326 452 catalog_manager.cc:4841] TS 0637b06d8e3c463cb5b1e5daad43f501 (127.21.43.67:37935): tablet 11784af0e54e4f9d93db40f98f6f7a0b (table test-table [id=4375bdc48ff4444984a0e759078d1824]) successfully deleted
W20260810 14:36:24.427795 454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:46117 (error 108)
I20260810 14:36:24.428017 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 688
I20260810 14:36:24.432045 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.431691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:46117 (local address 127.0.0.1:37890)
0810 14:36:24.431843 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:24.431932 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 14:36:24.432227 454 catalog_manager.cc:4584] TS a7fc2b9fa9fc4506827a1b84b50284dc (127.21.43.65:46117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11784af0e54e4f9d93db40f98f6f7a0b: Network error: Client connection negotiation failed: client connection to 127.21.43.65:46117: connect: Connection refused (error 111)
I20260810 14:36:24.437543 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 839
I20260810 14:36:24.446173 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 433
2026-08-10T14:36:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2653 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:24 dist-test-slave-chsk krb5kdc[1038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:24 dist-test-slave-chsk krb5kdc[1038](info): set up 2 sockets
Aug 10 14:36:24 dist-test-slave-chsk krb5kdc[1038](info): commencing operation
krb5kdc: starting...
W20260810 14:36:25.560554 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:36:25 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangeradmin_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangeradmin_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangerlookup_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangerlookup_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
I20260810 14:36:25.754472 21677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskA76VjH/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/postgres -l logfile start
2026-08-10 14:36:28.277 UTC [1070] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:36:28.278 UTC [1070] LOG: listening on IPv4 address "127.21.43.84", port 58809
2026-08-10 14:36:28.282 UTC [1070] LOG: listening on Unix socket "/tmp/.s.PGSQL.58809"
2026-08-10 14:36:28.305 UTC [1073] LOG: database system was shut down at 2026-08-10 14:36:26 UTC
2026-08-10 14:36:28.310 UTC [1070] LOG: database system is ready to accept connections
I20260810 14:36:29.317955 21677 mini_postgres.cc:96] Postgres bound to 58809
2026-08-10 14:36:29.325 UTC [1082] FATAL: database "slave" does not exist
127.21.43.84:58809 - accepting connections
I20260810 14:36:29.326215 21677 mini_ranger.cc:161] Starting Ranger...
I20260810 14:36:29.342779 21677 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:36:29.827562 21677 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:36:29.827659 21677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin
2026-08-10 14:36:30,117 [I] DB FLAVOR :POSTGRES
2026-08-10 14:36:30,118 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:36:30,118 [I] Checking connection..
2026-08-10 14:36:30,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:36:30,390 [I] Checking connection passed.
2026-08-10 14:36:30,391 [I] --------- Verifying version history table ---------
2026-08-10 14:36:30,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:36:30,651 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:36:30,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:36:30,924 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:36:30,925 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:36:30,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:36:31.195 UTC [1154] WARNING: there is no transaction in progress
2026-08-10 14:36:31,202 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:36:31,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:36:31,449 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:36:31,449 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:36:31,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:36:31,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:36:31,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-chsk.c.gcp-upstream.internal','N') ;"
2026-08-10 14:36:32,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:36:32,442 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:36:32,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:36:32,708 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:36:32,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:36:32,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:36:33,230 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:36:33,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:36:34.307 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.325 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.342 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.359 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.380 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.402 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.467 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.475 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.490 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.502 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.512 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.527 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.537 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.545 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.552 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:34.560 UTC [1281] WARNING: there is no transaction in progress
2026-08-10 14:36:35,217 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:36:35,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:36:35,468 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:36:35,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:36:35,748 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:36:35,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:36:36,014 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:36:36,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:36:36,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:36:36,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-chsk.c.gcp-upstream.internal';"
2026-08-10 14:36:36,740 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:36:36,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:36:37,011 [I] DB_PATCHES have already been applied
I20260810 14:36:37.017271 21677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskA76VjH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:36:37.017342 21677 mini_ranger.cc:193] Using host: 127.21.43.84
I20260810 14:36:37.021808 21677 mini_ranger.cc:227] Ranger admin URL: http://127.21.43.84:33299
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:36:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:36:38.210793 21677 mini_ranger.cc:160] Time spent starting Ranger: real 8.885s user 0.002s sys 0.011s
Aug 10, 2026 2:36:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:36:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 2:36:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:36:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:37:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:37:01.572048 21677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangerkms_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/rangerkms_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.43.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.43.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:37:01 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:37:01.692786 21677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:37:01.706149 21677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:37:02.633291 21677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:37:02.633404 21677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms
I20260810 14:37:02.633522 21677 mini_ranger_kms.cc:227] Using postgres at 127.21.43.84:58809
2026-08-10 14:37:03,406 [I] DB FLAVOR :POSTGRES
2026-08-10 14:37:03,407 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:37:03,407 [I] Checking connection
2026-08-10 14:37:03,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:37:03,681 [I] connection success
2026-08-10 14:37:03,682 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:37:03,682 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:37:03,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:37:03,965 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:37:03,966 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:37:03,966 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:37:03,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:58809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:37:04,282 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:37:04.930352 21677 mini_ranger_kms.cc:310] Created kms service
I20260810 14:37:05.125860 21677 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:37:05.182173 21677 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:37:05 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:05.617667 21677 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:37:05.617851 21677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:37:05.617880 21677 mini_ranger_kms.cc:240] Using host: 127.21.43.84
I20260810 14:37:05.622910 21677 mini_ranger_kms.cc:276] Ranger KMS PID: 1484
I20260810 14:37:05.623085 21677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.43.84:33941
Aug 10, 2026 2:37:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:37:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:37:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:37:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:37:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:37:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 14:37:07.990154 21677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.297s user 0.047s sys 0.455s
I20260810 14:37:07.990300 21677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:37:07.990417 21677 mini_ranger_kms.cc:401] 127.21.43.84:33941/kms/v1/keys
Aug 10 14:37:08 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 14:37:08 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
Aug 10 14:37:08 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
Aug 10 14:37:11 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:37:11.510769 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35551
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:35551
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:33941/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:11.688812 1542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:11.689114 1542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:11.689157 1542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:11.693558 1542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:11.693634 1542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:11.693655 1542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:11.693670 1542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:11.693686 1542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:11.699712 1542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.43.84:33941/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:35551
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:37:11.700899 1542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:11.702010 1542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:11.711382 1551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:11.711467 1548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:11.711536 1549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:11.711589 1542 server_base.cc:1034] running on GCE node
I20260810 14:37:11.712384 1542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:11.713562 1542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:11.714749 1542 hybrid_clock.cc:648] HybridClock initialized: now 1786372631714737 us; error 37 us; skew 500 ppm
Aug 10 14:37:11 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:11.722317 1542 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:37:11.723380 1542 webserver.cc:460] Webserver started at http://127.21.43.126:40345/ using document root <none> and password file <none>
I20260810 14:37:11.723964 1542 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:11.724063 1542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:11.724296 1542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:37:11 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372631, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:11.781014 1542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "3d1586d583ac4a10a65c4315a2f1c56a"
format_stamp: "Formatted at 2026-08-10 14:37:11 on dist-test-slave-chsk"
server_key: "d5468e91a4038a1706cf9b87cb32f05b"
server_key_iv: "f8829565d757be085579bd11bf9e2371"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:11.782106 1542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "3d1586d583ac4a10a65c4315a2f1c56a"
format_stamp: "Formatted at 2026-08-10 14:37:11 on dist-test-slave-chsk"
server_key: "d5468e91a4038a1706cf9b87cb32f05b"
server_key_iv: "f8829565d757be085579bd11bf9e2371"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:11.786988 1542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 14:37:11.803123 1559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:11.804737 1542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 14:37:11.804904 1542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "3d1586d583ac4a10a65c4315a2f1c56a"
format_stamp: "Formatted at 2026-08-10 14:37:11 on dist-test-slave-chsk"
server_key: "d5468e91a4038a1706cf9b87cb32f05b"
server_key_iv: "f8829565d757be085579bd11bf9e2371"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:11.805064 1542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:11.831123 1542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:11.834933 1542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:11.835243 1542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:11.852541 1542 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:35551
I20260810 14:37:11.852612 1616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:35551 every 8 connection(s)
I20260810 14:37:11.853868 1542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:11.857349 1617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:11.863267 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 1542
I20260810 14:37:11.863425 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:37:11.863725 21677 external_mini_cluster.cc:1428] Setting key ff6ca4bb8e29a03d2ce5b1ade118da71
I20260810 14:37:11.864945 1617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: Bootstrap starting.
I20260810 14:37:11.868995 1617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:11.870288 1617 log.cc:824] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:11.872866 1617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: No bootstrap required, opened a new log
Aug 10 14:37:11 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:37:11.878822 1617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } }
I20260810 14:37:11.879230 1617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:11.879324 1617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1586d583ac4a10a65c4315a2f1c56a, State: Initialized, Role: FOLLOWER
I20260810 14:37:11.879954 1617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } }
I20260810 14:37:11.880137 1617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:37:11.880218 1617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:37:11.880333 1617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:11.882661 1617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } }
I20260810 14:37:11.883291 1617 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d1586d583ac4a10a65c4315a2f1c56a; no voters:
I20260810 14:37:11.883795 1617 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:37:11.884270 1625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:11.884353 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.865376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.21.43.126:35551)
0810 14:37:11.865962 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.866012 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.866056 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:37:11.867574 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.867599 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.868022 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.868425 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:11.869398 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.871219 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.872213 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.872879 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.873060 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.876062 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:11.876070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:11.876089 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:11.876124 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:11.878593 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:11.880789 (+ 2196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:11.880795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:11.880803 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:11.880968 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:11.881365 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:11.881370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:11.881372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:11.882046 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:11.882428 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.882655 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.882979 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":141,"threads_started":1}
I20260810 14:37:11.884711 1625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [term 1 LEADER]: Becoming Leader. State: Replica: 3d1586d583ac4a10a65c4315a2f1c56a, State: Running, Role: LEADER
I20260810 14:37:11.885496 1617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:11.886129 1625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } }
I20260810 14:37:11.887506 1624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1586d583ac4a10a65c4315a2f1c56a. Latest consensus state: current_term: 1 leader_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } } }
I20260810 14:37:11.887666 1624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:11.887997 1627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1586d583ac4a10a65c4315a2f1c56a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35551 } } }
I20260810 14:37:11.888104 1627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:11.888581 1629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:11.894780 1629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:11.895401 1617 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:37:11.911121 1629 catalog_manager.cc:1347] Generated new cluster ID: c27711d4e685403eabce9ec853999b2b
I20260810 14:37:11.911307 1629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:11.920555 1629 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:11.922712 1629 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:11.938097 1629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: Generated new TSK 0
I20260810 14:37:11.939224 1629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:37:12.855028 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:33941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:13.056607 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:13.056993 1671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:13.057117 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:13.061483 1671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:13.061565 1671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:13.061655 1671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:37:13.067777 1671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.43.84:33941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:37:13.068943 1671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:13.070029 1671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:13.082185 1676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.082258 1677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.082415 1679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:13.082633 1671 server_base.cc:1034] running on GCE node
I20260810 14:37:13.083250 1671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:13.084172 1671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:13.087496 1671 hybrid_clock.cc:648] HybridClock initialized: now 1786372633087449 us; error 54 us; skew 500 ppm
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:13.092231 1671 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:37:13.093567 1671 webserver.cc:460] Webserver started at http://127.21.43.65:40155/ using document root <none> and password file <none>
I20260810 14:37:13.094439 1671 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:13.094627 1671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:13.095000 1671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:13.119359 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "2c4a54c45597464f9e0e502e538513bc"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "a19965ce29d8dcc7371a42f85a204de0"
server_key_iv: "d2c98dfd6eeb4af87823c58bdf557eb4"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.120178 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "2c4a54c45597464f9e0e502e538513bc"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "a19965ce29d8dcc7371a42f85a204de0"
server_key_iv: "d2c98dfd6eeb4af87823c58bdf557eb4"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.125947 1671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:37:13.149372 1688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.151130 1671 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260810 14:37:13.151304 1671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "2c4a54c45597464f9e0e502e538513bc"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "a19965ce29d8dcc7371a42f85a204de0"
server_key_iv: "d2c98dfd6eeb4af87823c58bdf557eb4"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.151445 1671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:13.166069 1671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:13.169651 1671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:13.169875 1671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:13.170692 1671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:37:13.171821 1671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:37:13.171893 1671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.171995 1671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:37:13.172034 1671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.190263 1671 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:36097
I20260810 14:37:13.190488 1821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:36097 every 8 connection(s)
I20260810 14:37:13.191757 1671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:37:13.199791 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 1671
I20260810 14:37:13.199931 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:37:13.200248 21677 external_mini_cluster.cc:1428] Setting key 8bb34fe403f2f6ed1d3068d2700a67ca
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:37:13.218055 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.194054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:53573 (local address 127.21.43.126:35551)
0810 14:37:13.194343 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:13.194348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:13.195519 (+ 1171us) server_negotiation.cc:407] Connection header received
0810 14:37:13.197231 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:13.197236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:13.197304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:13.197416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:13.199402 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.200304 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.206197 (+ 5893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.206779 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.206880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.210961 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:13.210966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:13.210969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:13.211000 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:13.213141 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.216076 (+ 2935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.216080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.216082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.216193 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.216669 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.216673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.216675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.216918 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:13.216997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:13.217725 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 14:37:13.217869 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20260810 14:37:13.220084 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.194571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35551 (local address 127.21.43.65:53573)
0810 14:37:13.195342 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.195400 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:13.196948 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:13.197612 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:13.197622 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:13.198197 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:13.199098 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.199120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.200965 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.200968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.205986 (+ 5018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.206007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.207190 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.207196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.207518 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.208366 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:13.208401 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:13.210725 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:13.213298 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.213305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.213325 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.213719 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.216324 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.216327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.216330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.216552 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.217072 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:13.217080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:13.217437 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 14:37:13.218569 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0810 14:37:13.218940 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":228,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:37:13.228962 1822 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35551
I20260810 14:37:13.229256 1822 heartbeater.cc:464] Registering TS with master...
I20260810 14:37:13.229905 1822 heartbeater.cc:511] Master 127.21.43.126:35551 requested a full tablet report, sending...
I20260810 14:37:13.231961 1578 ts_manager.cc:194] Registered new tserver with Master: 2c4a54c45597464f9e0e502e538513bc (127.21.43.65:36097)
I20260810 14:37:13.233713 1578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:53573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:37:13.269848 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:33941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:37:13.483614 1831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:13.483966 1831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:13.484031 1831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:13.490432 1831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:13.490640 1831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:13.490754 1831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:37:13.499541 1831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.43.84:33941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:37:13.501037 1831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:13.502735 1831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:13.517326 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.517549 1840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.517393 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:13.522293 1831 server_base.cc:1034] running on GCE node
I20260810 14:37:13.523021 1831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:13.524116 1831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:13.525449 1831 hybrid_clock.cc:648] HybridClock initialized: now 1786372633525429 us; error 37 us; skew 500 ppm
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:13.530262 1831 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:37:13.531687 1831 webserver.cc:460] Webserver started at http://127.21.43.66:39469/ using document root <none> and password file <none>
I20260810 14:37:13.532405 1831 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:13.532487 1831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:13.532709 1831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:13.549638 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "db4a6f442c5043a29c20ae3b42b78fb2"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "96863644c2b6cfcc053a3e9c741f6f23"
server_key_iv: "8751b58484344d64b13d6503a9f83d5c"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.550290 1831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "db4a6f442c5043a29c20ae3b42b78fb2"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "96863644c2b6cfcc053a3e9c741f6f23"
server_key_iv: "8751b58484344d64b13d6503a9f83d5c"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.556911 1831 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 14:37:13.574141 1848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.575893 1831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:37:13.576059 1831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "db4a6f442c5043a29c20ae3b42b78fb2"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "96863644c2b6cfcc053a3e9c741f6f23"
server_key_iv: "8751b58484344d64b13d6503a9f83d5c"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:13.576398 1831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:13.599805 1831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:13.603833 1831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:13.604128 1831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:13.604871 1831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:37:13.606102 1831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:37:13.606215 1831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.606319 1831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:37:13.606382 1831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.633780 1831 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:46467
I20260810 14:37:13.635213 1831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:37:13.636013 1983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:46467 every 8 connection(s)
I20260810 14:37:13.643179 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 1831
I20260810 14:37:13.643343 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:37:13.643674 21677 external_mini_cluster.cc:1428] Setting key bcac1c6ee89ce5e62f1014b65e354509
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:37:13.668960 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.639448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48393 (local address 127.21.43.126:35551)
0810 14:37:13.640664 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:13.640670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:13.641277 (+ 607us) server_negotiation.cc:407] Connection header received
0810 14:37:13.649140 (+ 7863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:13.649144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:13.649212 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:13.649277 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:13.651600 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.652465 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.654561 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.654942 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.655048 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.661172 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:13.661178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:13.661189 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:13.661223 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:13.663344 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.664081 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.664085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.664087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.664177 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.667350 (+ 3173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.667355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.667357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.667621 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:13.667714 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:13.668599 (+ 885us) server_negotiation.cc:299] Negotiation successful
0810 14:37:13.668776 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074}
I20260810 14:37:13.671257 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.639301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35551 (local address 127.21.43.66:48393)
0810 14:37:13.641077 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.641151 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:13.648799 (+ 7648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:13.649804 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:13.649816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:13.650421 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:13.651397 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.651420 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.652685 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.652688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.654359 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.654371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.655338 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.655343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.655636 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.656524 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:13.656571 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:13.660865 (+ 4294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:13.663529 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.663535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.663566 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.663954 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.664289 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.664292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.664294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.664500 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.667813 (+ 3313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:13.667824 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:13.668240 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 14:37:13.669549 (+ 1309us) client_negotiation.cc:240] Negotiation successful
0810 14:37:13.669978 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":294,"threads_started":1}
I20260810 14:37:13.676823 1984 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35551
I20260810 14:37:13.677110 1984 heartbeater.cc:464] Registering TS with master...
I20260810 14:37:13.677824 1984 heartbeater.cc:511] Master 127.21.43.126:35551 requested a full tablet report, sending...
I20260810 14:37:13.679286 1578 ts_manager.cc:194] Registered new tserver with Master: db4a6f442c5043a29c20ae3b42b78fb2 (127.21.43.66:46467)
I20260810 14:37:13.680191 1578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:48393
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:37:13.720715 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:33941/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:13.929839 1990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:13.930356 1990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:13.930492 1990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:13.936805 1990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:13.936975 1990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:13.937127 1990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:37:13.945598 1990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.43.84:33941/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:37:13.947464 1990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:13.949002 1990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:13.963829 1996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.963832 1995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.964485 1998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:13.965631 1990 server_base.cc:1034] running on GCE node
I20260810 14:37:13.966193 1990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:13.967134 1990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:13.968442 1990 hybrid_clock.cc:648] HybridClock initialized: now 1786372633968414 us; error 41 us; skew 500 ppm
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:13.973335 1990 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:37:13.974633 1990 webserver.cc:460] Webserver started at http://127.21.43.67:37003/ using document root <none> and password file <none>
I20260810 14:37:13.975481 1990 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:13.975639 1990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:13.975952 1990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:37:13 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:13.999680 1990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "f878154843c14580b0f59a231a49750f"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "f3dd8aa05d89e0ae230ac51fe945fadd"
server_key_iv: "75a8374b15cbc665c48e17349c1edf2e"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:14.000759 1990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "f878154843c14580b0f59a231a49750f"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "f3dd8aa05d89e0ae230ac51fe945fadd"
server_key_iv: "75a8374b15cbc665c48e17349c1edf2e"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:14.007233 1990 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260810 14:37:14.035601 2005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:14.037438 1990 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260810 14:37:14.037669 1990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "f878154843c14580b0f59a231a49750f"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-chsk"
server_key: "f3dd8aa05d89e0ae230ac51fe945fadd"
server_key_iv: "75a8374b15cbc665c48e17349c1edf2e"
server_key_version: "kuduclusterkey@0"
I20260810 14:37:14.037914 1990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:14.076676 1990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:14.080654 1990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:14.080924 1990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:14.082007 1990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:37:14.083281 1990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:37:14.083401 1990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:14.083503 1990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:37:14.083566 1990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:37:14 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:14.109678 1990 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:45899
I20260810 14:37:14.111371 1990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:37:14.111449 2131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:45899 every 8 connection(s)
I20260810 14:37:14.113940 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 1990
I20260810 14:37:14.114070 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:37:14.114377 21677 external_mini_cluster.cc:1428] Setting key d9f7a08a77a3ca840920ef35c36fd0f7
Aug 10 14:37:14 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:37:14.134615 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.115056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:41639 (local address 127.21.43.126:35551)
0810 14:37:14.115250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.115256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.116393 (+ 1137us) server_negotiation.cc:407] Connection header received
0810 14:37:14.118436 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.118448 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.118509 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.118564 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.120985 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.121791 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.123693 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.124046 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.124131 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.128044 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.128048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.128051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.128082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.129962 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.132527 (+ 2565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.132531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.132533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.132641 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.133198 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.133202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.133204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.133459 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.133552 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.134315 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.134454 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:37:14.136534 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.115558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35551 (local address 127.21.43.67:41639)
0810 14:37:14.116234 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.116291 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.118181 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.119002 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.119013 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.119593 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.120809 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.120833 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.121963 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.121966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.123530 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.123542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.124365 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.124370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.124634 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.125519 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.125541 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.127825 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.131959 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.131966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.131985 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.132392 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.132788 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.132791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.132793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.133071 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.133626 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.133633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.134003 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.135037 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.135426 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":219,"threads_started":1}
I20260810 14:37:14.138001 2132 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35551
I20260810 14:37:14.138259 2132 heartbeater.cc:464] Registering TS with master...
I20260810 14:37:14.138778 2132 heartbeater.cc:511] Master 127.21.43.126:35551 requested a full tablet report, sending...
I20260810 14:37:14.140126 1578 ts_manager.cc:194] Registered new tserver with Master: f878154843c14580b0f59a231a49750f (127.21.43.67:45899)
I20260810 14:37:14.140717 1578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:41639
I20260810 14:37:14.143702 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:37:14.161105 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.147612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.21.43.126:35551)
0810 14:37:14.147735 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.147739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.150241 (+ 2502us) server_negotiation.cc:407] Connection header received
0810 14:37:14.150400 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.150403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.150455 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.150516 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.151681 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.152466 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.153705 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.154031 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.154115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.156228 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.156232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.156235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.156265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.158812 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.159524 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.159528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.159530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.159684 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.160058 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.160061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.160063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.160309 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.160423 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.160791 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.160932 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:37:14.165454 1578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:37:14.170055 1578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:37:14.203955 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.186766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36097 (local address 127.0.0.1:45654)
0810 14:37:14.195079 (+ 8313us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.195135 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.198008 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.198497 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.198503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.198525 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:37:14.198785 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.198793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.200529 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.200535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.202076 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.202087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.203424 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.203427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.203547 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.203586 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.203693 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.203805 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8092,"thread_start_us":244,"threads_started":1}
I20260810 14:37:14.204850 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.187608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.21.43.65:36097)
0810 14:37:14.193325 (+ 5717us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.193333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.198105 (+ 4772us) server_negotiation.cc:407] Connection header received
0810 14:37:14.198180 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.198185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.198341 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.198497 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:37:14.198941 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.200392 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.202224 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.203084 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.203207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.203359 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.204606 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.204708 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5573,"thread_start_us":204,"threads_started":1}
I20260810 14:37:14.208056 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.189742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:46467 (local address 127.0.0.1:41306)
0810 14:37:14.195248 (+ 5506us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.195260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.195437 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.198732 (+ 3295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.198745 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.198762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:37:14.198906 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.198912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.204267 (+ 5355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.204269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.206138 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.206147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.207469 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.207473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.207691 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.207750 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.207823 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.207912 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"mutex_wait_us":81,"thread_start_us":212,"threads_started":1}
I20260810 14:37:14.208181 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.192536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.21.43.67:45899)
0810 14:37:14.194793 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.194831 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.196634 (+ 1803us) server_negotiation.cc:407] Connection header received
0810 14:37:14.197426 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.197438 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.197648 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.197805 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:37:14.201362 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.202688 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.206332 (+ 3644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.207112 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.207280 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.207411 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.207920 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.208016 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2156,"thread_start_us":236,"threads_started":1}
I20260810 14:37:14.208693 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.191977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45899 (local address 127.0.0.1:47184)
0810 14:37:14.196404 (+ 4427us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.196418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.196540 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.199029 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.199031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.199060 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:37:14.199177 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.199184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.202979 (+ 3795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.202982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.204096 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.204110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.207469 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.207473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.207697 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.207748 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.207814 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.208561 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4359,"mutex_wait_us":503,"thread_start_us":184,"threads_started":1}
I20260810 14:37:14.208781 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.190319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.21.43.66:46467)
0810 14:37:14.194393 (+ 4074us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.194399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.195612 (+ 1213us) server_negotiation.cc:407] Connection header received
0810 14:37:14.195700 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.195707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.195949 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.196095 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:37:14.199317 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.200476 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.206274 (+ 5798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.207166 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.207269 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.207411 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.208450 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.208602 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3967,"thread_start_us":183,"threads_started":1}
I20260810 14:37:14.208702 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 01f437195a344a25bf95d89077b33083 (DEFAULT_TABLE table=test-table [id=1bfff20330e349dcbaaf6647ffc61474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:37:14.210111 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f437195a344a25bf95d89077b33083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:14.211453 1891 tablet_service.cc:1467] Processing CreateTablet for tablet 01f437195a344a25bf95d89077b33083 (DEFAULT_TABLE table=test-table [id=1bfff20330e349dcbaaf6647ffc61474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:37:14.212608 1891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f437195a344a25bf95d89077b33083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:14.220363 2154 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Bootstrap starting.
I20260810 14:37:14.223237 2154 tablet_bootstrap.cc:654] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.224746 2154 log.cc:824] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.225764 2155 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Bootstrap starting.
I20260810 14:37:14.226594 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 01f437195a344a25bf95d89077b33083 (DEFAULT_TABLE table=test-table [id=1bfff20330e349dcbaaf6647ffc61474]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:37:14.227798 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f437195a344a25bf95d89077b33083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:14.229226 2155 tablet_bootstrap.cc:654] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.230346 2155 log.cc:824] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.232964 2154 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: No bootstrap required, opened a new log
I20260810 14:37:14.233251 2154 ts_tablet_manager.cc:1397] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260810 14:37:14.235808 2155 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: No bootstrap required, opened a new log
I20260810 14:37:14.236125 2155 ts_tablet_manager.cc:1397] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260810 14:37:14.237227 1822 heartbeater.cc:503] Master 127.21.43.126:35551 was elected leader, sending a full tablet report...
I20260810 14:37:14.239317 2154 raft_consensus.cc:348] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.240051 2154 raft_consensus.cc:374] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:14.240166 2154 raft_consensus.cc:729] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4a54c45597464f9e0e502e538513bc, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.240747 2154 consensus_queue.cc:260] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.242134 2154 ts_tablet_manager.cc:1428] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260810 14:37:14.244757 2161 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Bootstrap starting.
I20260810 14:37:14.244673 2155 raft_consensus.cc:348] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.245179 2155 raft_consensus.cc:374] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:14.245289 2155 raft_consensus.cc:729] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4a6f442c5043a29c20ae3b42b78fb2, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.246001 2155 consensus_queue.cc:260] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.247090 2155 ts_tablet_manager.cc:1428] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260810 14:37:14.247501 1984 heartbeater.cc:503] Master 127.21.43.126:35551 was elected leader, sending a full tablet report...
I20260810 14:37:14.248621 2161 tablet_bootstrap.cc:654] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.250226 2161 log.cc:824] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.252547 2161 tablet_bootstrap.cc:492] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: No bootstrap required, opened a new log
I20260810 14:37:14.252787 2161 ts_tablet_manager.cc:1397] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:37:14.258183 2161 raft_consensus.cc:348] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.258477 2161 raft_consensus.cc:374] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:14.258576 2161 raft_consensus.cc:729] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f878154843c14580b0f59a231a49750f, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.259060 2161 consensus_queue.cc:260] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.259915 2132 heartbeater.cc:503] Master 127.21.43.126:35551 was elected leader, sending a full tablet report...
I20260810 14:37:14.260234 2161 ts_tablet_manager.cc:1428] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260810 14:37:14.368165 2133 tablet.cc:2369] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:37:14.387688 1985 tablet.cc:2369] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:37:14 dist-test-slave-chsk krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
W20260810 14:37:14.458921 1823 tablet.cc:2369] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:37:14.490067 2163 raft_consensus.cc:482] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:37:14.490362 2163 raft_consensus.cc:504] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.491876 2163 leader_election.cc:288] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c4a54c45597464f9e0e502e538513bc (127.21.43.65:36097), f878154843c14580b0f59a231a49750f (127.21.43.67:45899)
I20260810 14:37:14.499838 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.492742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48313 (local address 127.21.43.65:36097)
0810 14:37:14.492882 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.492888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.494022 (+ 1134us) server_negotiation.cc:407] Connection header received
0810 14:37:14.494248 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.494252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.494360 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.494415 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:37:14.495176 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.496031 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.498016 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.498780 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.498907 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.498973 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.499575 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.499679 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:37:14.500751 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.492094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36097 (local address 127.21.43.66:48313)
0810 14:37:14.493915 (+ 1821us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.493934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.494116 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.494591 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.494594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.494616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:37:14.495005 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.495014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.496199 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.496203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.497842 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.497853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.499191 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.499195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.499343 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.499385 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.500308 (+ 923us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.500405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"thread_start_us":1154,"threads_started":1}
I20260810 14:37:14.501996 1762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f437195a344a25bf95d89077b33083" candidate_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4a54c45597464f9e0e502e538513bc" is_pre_election: true
I20260810 14:37:14.502369 1762 raft_consensus.cc:2454] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db4a6f442c5043a29c20ae3b42b78fb2 in term 0.
I20260810 14:37:14.503224 1851 leader_election.cc:302] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4a54c45597464f9e0e502e538513bc, db4a6f442c5043a29c20ae3b42b78fb2; no voters:
I20260810 14:37:14.503600 2163 raft_consensus.cc:2781] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:37:14.503700 2163 raft_consensus.cc:482] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:37:14.503756 2163 raft_consensus.cc:3037] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.505767 2163 raft_consensus.cc:504] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.508810 2163 leader_election.cc:288] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 2c4a54c45597464f9e0e502e538513bc (127.21.43.65:36097), f878154843c14580b0f59a231a49750f (127.21.43.67:45899)
I20260810 14:37:14.514436 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.492485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:56831 (local address 127.21.43.67:45899)
0810 14:37:14.492631 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.492636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.501559 (+ 8923us) server_negotiation.cc:407] Connection header received
0810 14:37:14.501795 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.501798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.501857 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.501909 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:37:14.509573 (+ 7664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.510327 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.512064 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.512718 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.512824 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.512896 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.513296 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.513386 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:37:14.514981 1762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f437195a344a25bf95d89077b33083" candidate_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4a54c45597464f9e0e502e538513bc"
I20260810 14:37:14.515244 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.492158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45899 (local address 127.21.43.66:56831)
0810 14:37:14.501476 (+ 9318us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.501491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.501657 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.509232 (+ 7575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.509237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.509271 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:37:14.509380 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.509387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.510530 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.510535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.511867 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.511879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.513044 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.513048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.513131 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.513158 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.514857 (+ 1699us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.514968 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9260,"thread_start_us":1349,"threads_started":1}
I20260810 14:37:14.515199 1762 raft_consensus.cc:3037] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.516232 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f437195a344a25bf95d89077b33083" candidate_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f878154843c14580b0f59a231a49750f" is_pre_election: true
I20260810 14:37:14.516353 2083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f437195a344a25bf95d89077b33083" candidate_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f878154843c14580b0f59a231a49750f"
I20260810 14:37:14.516609 2083 raft_consensus.cc:3037] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.517333 1762 raft_consensus.cc:2454] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db4a6f442c5043a29c20ae3b42b78fb2 in term 1.
I20260810 14:37:14.517841 1851 leader_election.cc:302] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c4a54c45597464f9e0e502e538513bc, db4a6f442c5043a29c20ae3b42b78fb2; no voters:
I20260810 14:37:14.518126 2163 raft_consensus.cc:2781] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:14.518322 2083 raft_consensus.cc:2454] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db4a6f442c5043a29c20ae3b42b78fb2 in term 1.
I20260810 14:37:14.518903 2163 raft_consensus.cc:686] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 1 LEADER]: Becoming Leader. State: Replica: db4a6f442c5043a29c20ae3b42b78fb2, State: Running, Role: LEADER
I20260810 14:37:14.519819 2163 consensus_queue.cc:237] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } }
I20260810 14:37:14.525496 1578 catalog_manager.cc:5495] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to db4a6f442c5043a29c20ae3b42b78fb2 (127.21.43.66). New cstate: current_term: 1 leader_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db4a6f442c5043a29c20ae3b42b78fb2" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 } health_report { overall_health: UNKNOWN } } }
I20260810 14:37:14.568235 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.562940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.21.43.66:46467)
0810 14:37:14.563082 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.563089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.563108 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:37:14.563380 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.563383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.563454 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.563550 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:14.563943 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.564683 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.565919 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.566252 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.566360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.566823 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:14.567444 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:14.567570 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.568017 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.568083 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:37:14.576810 1762 raft_consensus.cc:1261] T 01f437195a344a25bf95d89077b33083 P 2c4a54c45597464f9e0e502e538513bc [term 1 FOLLOWER]: Refusing update from remote peer db4a6f442c5043a29c20ae3b42b78fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:14.577875 2083 raft_consensus.cc:1261] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 1 FOLLOWER]: Refusing update from remote peer db4a6f442c5043a29c20ae3b42b78fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:14.578576 2163 consensus_queue.cc:1035] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4a54c45597464f9e0e502e538513bc" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:14.579581 2163 consensus_queue.cc:1035] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f878154843c14580b0f59a231a49750f" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:14.607793 1578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:37:14.608023 1578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:37:14.611014 1578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1586d583ac4a10a65c4315a2f1c56a: Sending DeleteTablet for 3 replicas of tablet 01f437195a344a25bf95d89077b33083
I20260810 14:37:14.612185 1726 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f437195a344a25bf95d89077b33083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:37:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:45654
I20260810 14:37:14.613111 2059 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f437195a344a25bf95d89077b33083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:37:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:47184
I20260810 14:37:14.614187 1891 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f437195a344a25bf95d89077b33083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:37:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41306
I20260810 14:37:14.614696 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 1671
I20260810 14:37:14.614749 2177 mvcc.cc:205] Tried to move back new op lower bound from 7316982311213543424 to 7316982310996451328. Current Snapshot: MvccSnapshot[applied={T|T < 7316982311213543424}]
I20260810 14:37:14.619477 2183 tablet_replica.cc:330] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: stopping tablet replica
I20260810 14:37:14.620011 2183 raft_consensus.cc:2229] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:37:14.620446 2183 pending_rounds.cc:62] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Trying to abort 2 pending ops.
I20260810 14:37:14.620615 2183 pending_rounds.cc:68] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316982311213543424 op_type: WRITE_OP write_request { tablet_id: "01f437195a344a25bf95d89077b33083" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\306\306\347\323\006\"8\n\ntest-admin\022*\n 1bfff20330e349dcbaaf6647ffc61474\020\001\030\001 \001(\001" signature: "<redacted>""+2\273\214\371\247\002\253g\001\215?\246\006\232\256eW\024*\330\013\332S\337bx{}\213v\037\367~\364;\235\365\243\236\021\336\266\350\332\237\215n\346\nW\303\375\007\233\262\220k\243\236\302A\314(" signing_key_seq_num: 0 } } request_id { client_id: "5fd9d60c5de24a14a7ffc364c26ea283" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260810 14:37:14.621905 2183 pending_rounds.cc:68] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316982310996451328 op_type: NO_OP noop_request { }
I20260810 14:37:14.622141 2183 raft_consensus.cc:2866] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 14:37:14.622305 2183 raft_consensus.cc:2258] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 14:37:14.622400 1849 consensus_peers.cc:487] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 -> Peer f878154843c14580b0f59a231a49750f (127.21.43.67:45899): Couldn't send request to peer f878154843c14580b0f59a231a49750f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:37:14.624568 2183 ts_tablet_manager.cc:1905] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:37:14.624922 2184 tablet_replica.cc:330] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: stopping tablet replica
I20260810 14:37:14.628095 2184 raft_consensus.cc:2229] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:37:14.631691 2184 raft_consensus.cc:2258] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:37:14.638099 2184 ts_tablet_manager.cc:1905] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:37:14.641167 2184 ts_tablet_manager.cc:1918] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:37:14.642115 2183 ts_tablet_manager.cc:1918] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:37:14.642310 2183 log.cc:1196] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/01f437195a344a25bf95d89077b33083
I20260810 14:37:14.642742 2183 ts_tablet_manager.cc:1939] T 01f437195a344a25bf95d89077b33083 P f878154843c14580b0f59a231a49750f: Deleting consensus metadata
I20260810 14:37:14.644126 2184 log.cc:1196] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/01f437195a344a25bf95d89077b33083
I20260810 14:37:14.644788 1560 catalog_manager.cc:4841] TS f878154843c14580b0f59a231a49750f (127.21.43.67:45899): tablet 01f437195a344a25bf95d89077b33083 (table test-table [id=1bfff20330e349dcbaaf6647ffc61474]) successfully deleted
I20260810 14:37:14.645349 2184 ts_tablet_manager.cc:1939] T 01f437195a344a25bf95d89077b33083 P db4a6f442c5043a29c20ae3b42b78fb2: Deleting consensus metadata
I20260810 14:37:14.646802 1564 catalog_manager.cc:4841] TS db4a6f442c5043a29c20ae3b42b78fb2 (127.21.43.66:46467): tablet 01f437195a344a25bf95d89077b33083 (table test-table [id=1bfff20330e349dcbaaf6647ffc61474]) successfully deleted
W20260810 14:37:14.647526 1563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:36097 (error 108)
I20260810 14:37:14.648588 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 1831
I20260810 14:37:14.653492 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.653132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:36097 (local address 127.0.0.1:45660)
0810 14:37:14.653295 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.653391 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:37:14.653725 1563 catalog_manager.cc:4584] TS 2c4a54c45597464f9e0e502e538513bc (127.21.43.65:36097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01f437195a344a25bf95d89077b33083: Network error: Client connection negotiation failed: client connection to 127.21.43.65:36097: connect: Connection refused (error 111)
I20260810 14:37:14.662533 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 1990
I20260810 14:37:14.679654 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 1542
I20260810 14:37:14.690263 21677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:37:15.035845 21677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:37:15.036000 21677 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:37:15.381454 21677 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:37:15.381 UTC [1070] LOG: received smart shutdown request
2026-08-10 14:37:15.385 UTC [1070] LOG: background worker "logical replication launcher" (PID 1080) exited with exit code 1
2026-08-10 14:37:15.386 UTC [1074] LOG: shutting down
2026-08-10 14:37:15.564 UTC [1070] LOG: database system is shut down
2026-08-10T14:37:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51245 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:15 dist-test-slave-chsk krb5kdc[2197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:15 dist-test-slave-chsk krb5kdc[2197](info): set up 2 sockets
Aug 10 14:37:15 dist-test-slave-chsk krb5kdc[2197](info): commencing operation
krb5kdc: starting...
W20260810 14:37:16.789688 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:37:16 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangeradmin_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangeradmin_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangerlookup_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangerlookup_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
I20260810 14:37:16.973449 21677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskA76VjH/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/postgres -l logfile start
2026-08-10 14:37:19.476 UTC [2231] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:37:19.476 UTC [2231] LOG: listening on IPv4 address "127.21.43.84", port 38059
2026-08-10 14:37:19.480 UTC [2231] LOG: listening on Unix socket "/tmp/.s.PGSQL.38059"
2026-08-10 14:37:19.517 UTC [2234] LOG: database system was shut down at 2026-08-10 14:37:18 UTC
2026-08-10 14:37:19.521 UTC [2231] LOG: database system is ready to accept connections
I20260810 14:37:20.532919 21677 mini_postgres.cc:96] Postgres bound to 38059
2026-08-10 14:37:20.540 UTC [2244] FATAL: database "slave" does not exist
127.21.43.84:38059 - accepting connections
I20260810 14:37:20.541589 21677 mini_ranger.cc:161] Starting Ranger...
I20260810 14:37:20.558521 21677 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:37:21.037675 21677 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:37:21.037775 21677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin
2026-08-10 14:37:21,325 [I] DB FLAVOR :POSTGRES
2026-08-10 14:37:21,326 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:37:21,326 [I] Checking connection..
2026-08-10 14:37:21,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:37:21,605 [I] Checking connection passed.
2026-08-10 14:37:21,606 [I] --------- Verifying version history table ---------
2026-08-10 14:37:21,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:37:21,867 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:37:21,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:37:22,115 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:37:22,115 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:37:22,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:37:22.386 UTC [2321] WARNING: there is no transaction in progress
2026-08-10 14:37:22,394 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:37:22,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:37:22,676 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:37:22,677 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:37:22,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:37:22,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:37:23,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-chsk.c.gcp-upstream.internal','N') ;"
2026-08-10 14:37:23,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:37:23,672 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:37:23,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:37:23,919 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:37:23,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:37:24,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:37:24,475 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:37:24,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:37:25.482 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.499 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.517 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.533 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.550 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.567 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.623 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.630 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.645 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.655 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.666 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.680 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.690 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.697 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.704 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:25.711 UTC [2450] WARNING: there is no transaction in progress
2026-08-10 14:37:26,358 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:37:26,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:37:26,636 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:37:26,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:37:26,886 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:37:26,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:37:27,171 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:37:27,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:37:27,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:37:27,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-chsk.c.gcp-upstream.internal';"
2026-08-10 14:37:27,975 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:37:27,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:37:28,227 [I] DB_PATCHES have already been applied
I20260810 14:37:28.233188 21677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskA76VjH/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:37:28.233259 21677 mini_ranger.cc:193] Using host: 127.21.43.84
I20260810 14:37:28.237629 21677 mini_ranger.cc:227] Ranger admin URL: http://127.21.43.84:47055
Aug 10, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:37:29.433286 21677 mini_ranger.cc:160] Time spent starting Ranger: real 8.892s user 0.000s sys 0.012s
Aug 10, 2026 2:37:49 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:37:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 2:37:49 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:37:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:37:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:37:51.652931 21677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.43.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.43.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangerkms_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.43.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/rangerkms_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.43.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.43.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.43.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/HTTP_127.21.43.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:37:51 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372671, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:37:51.749089 21677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:37:51.761590 21677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:37:52.634799 21677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:37:52.634900 21677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms
I20260810 14:37:52.634919 21677 mini_ranger_kms.cc:227] Using postgres at 127.21.43.84:38059
2026-08-10 14:37:52,860 [I] DB FLAVOR :POSTGRES
2026-08-10 14:37:52,860 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:37:52,861 [I] Checking connection
2026-08-10 14:37:52,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:37:53,131 [I] connection success
2026-08-10 14:37:53,131 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:37:53,131 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:37:53,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:37:53,462 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:37:53,462 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:37:53,462 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:37:53,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.43.84:38059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:37:53,768 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:37:54.307865 21677 mini_ranger_kms.cc:310] Created kms service
I20260810 14:37:54.550446 21677 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:37:54.597465 21677 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:37:54 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372671, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:37:55.078301 21677 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:37:55.078462 21677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms:/tmp/dist-test-taskA76VjH/build/debug/bin/postgresql.jar:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskA76VjH/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:37:55.078493 21677 mini_ranger_kms.cc:240] Using host: 127.21.43.84
I20260810 14:37:55.084223 21677 mini_ranger_kms.cc:276] Ranger KMS PID: 2645
I20260810 14:37:55.084319 21677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.43.84:60235
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 14:37:57.547200 21677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.798s user 0.008s sys 0.202s
I20260810 14:37:57.547344 21677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:37:57.547421 21677 mini_ranger_kms.cc:401] 127.21.43.84:60235/kms/v1/keys
Aug 10 14:37:57 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372677, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 14:37:58 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372677, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
Aug 10 14:37:58 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372677, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.43.84@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
Aug 10 14:38:00 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:38:00.855376 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38451
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38451
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:00.978667 2703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:00.979161 2703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:00.979300 2703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:00.983741 2703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:00.983806 2703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:00.983829 2703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:00.983845 2703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:00.983896 2703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:00.989974 2703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.43.84:60235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38451
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:00.991268 2703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:00.992393 2703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:01.002038 2710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:01.002105 2703 server_base.cc:1034] running on GCE node
W20260810 14:38:01.002324 2713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:01.002040 2709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:01.003113 2703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:01.004524 2703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:01.005728 2703 hybrid_clock.cc:648] HybridClock initialized: now 1786372681005708 us; error 43 us; skew 500 ppm
Aug 10 14:38:01 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372681, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:01.013681 2703 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:01.014878 2703 webserver.cc:460] Webserver started at http://127.21.43.126:35573/ using document root <none> and password file <none>
I20260810 14:38:01.015555 2703 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:01.015620 2703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:01.015870 2703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:38:01 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372681, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:01.067790 2703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "a3dd1dd8042643a5a2e5a3e300319ce5"
format_stamp: "Formatted at 2026-08-10 14:38:01 on dist-test-slave-chsk"
server_key: "96a74ff9da2ceffcfe5ef2e9ce648629"
server_key_iv: "6a858087218ad963042d30284eeb25e1"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:01.068462 2703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "a3dd1dd8042643a5a2e5a3e300319ce5"
format_stamp: "Formatted at 2026-08-10 14:38:01 on dist-test-slave-chsk"
server_key: "96a74ff9da2ceffcfe5ef2e9ce648629"
server_key_iv: "6a858087218ad963042d30284eeb25e1"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:01.073081 2703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:38:01.089540 2720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:01.091077 2703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:01.091296 2703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "a3dd1dd8042643a5a2e5a3e300319ce5"
format_stamp: "Formatted at 2026-08-10 14:38:01 on dist-test-slave-chsk"
server_key: "96a74ff9da2ceffcfe5ef2e9ce648629"
server_key_iv: "6a858087218ad963042d30284eeb25e1"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:01.091457 2703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:01.115809 2703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:01.119130 2703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:01.119446 2703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:01.126739 2703 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38451
I20260810 14:38:01.126783 2775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38451 every 8 connection(s)
I20260810 14:38:01.127856 2703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:01.131754 2776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:01.135475 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 2703
I20260810 14:38:01.135655 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:38:01.135982 21677 external_mini_cluster.cc:1428] Setting key bc8d65d3f006c5d6d474d8c3e44eac03
I20260810 14:38:01.139963 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Bootstrap starting.
I20260810 14:38:01.143261 2776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:01.144313 2776 log.cc:824] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:38:01 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:01.147199 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: No bootstrap required, opened a new log
I20260810 14:38:01.151784 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:01.137569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.21.43.126:38451)
0810 14:38:01.138028 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:01.138045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:01.138083 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:38:01.138731 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:01.138748 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:01.139087 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:01.139503 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:01.140405 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:01.141396 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:01.142322 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:01.142964 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:01.143142 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:01.146093 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:01.146099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:01.146114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:01.146142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:01.148596 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:01.149094 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:01.149099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:01.149104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:01.149237 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:01.149532 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:01.149537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:01.149539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:01.150083 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:01.150316 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:01.150622 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 14:38:01.150949 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260810 14:38:01.151618 2776 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:01.151921 2776 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:01.152024 2776 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dd1dd8042643a5a2e5a3e300319ce5, State: Initialized, Role: FOLLOWER
I20260810 14:38:01.152562 2776 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:01.152745 2776 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:01.152848 2776 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:01.152967 2776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:01.154348 2776 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:01.154784 2776 leader_election.cc:302] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3dd1dd8042643a5a2e5a3e300319ce5; no voters:
I20260810 14:38:01.155251 2776 leader_election.cc:288] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:38:01.155342 2781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:01.155614 2781 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 LEADER]: Becoming Leader. State: Replica: a3dd1dd8042643a5a2e5a3e300319ce5, State: Running, Role: LEADER
I20260810 14:38:01.156016 2781 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:01.156792 2776 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:01.158015 2782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } } }
I20260810 14:38:01.158167 2782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:01.158594 2791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:01.158594 2783 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3dd1dd8042643a5a2e5a3e300319ce5. Latest consensus state: current_term: 1 leader_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } } }
I20260810 14:38:01.158726 2783 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:01.161809 2791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:01.162372 2776 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:38:01.172957 2791 catalog_manager.cc:1347] Generated new cluster ID: f6a8debf37f34a0da5747613ac727bf4
I20260810 14:38:01.173158 2791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:01.190295 2791 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:38:01.191730 2791 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:01.208343 2791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Generated new TSK 0
I20260810 14:38:01.209533 2791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:38:02.021219 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:02.213493 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:02.213937 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:02.214035 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:02.220657 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:02.220832 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:02.220969 2824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:02.229964 2824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:02.231565 2824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:02.232724 2824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:02.247349 2837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:02.247509 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:02.248206 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:02.248881 2824 server_base.cc:1034] running on GCE node
I20260810 14:38:02.249410 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:02.250195 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:02.251374 2824 hybrid_clock.cc:648] HybridClock initialized: now 1786372682251337 us; error 53 us; skew 500 ppm
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:02.255929 2824 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:02.257254 2824 webserver.cc:460] Webserver started at http://127.21.43.65:38913/ using document root <none> and password file <none>
I20260810 14:38:02.257994 2824 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:02.258095 2824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:02.258385 2824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:02.278000 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "c193eb76023243f589ffb5395d88ffa5"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "c95ad08251e35e48d4c012bea28d185b"
server_key_iv: "fd400ccf1a80b065800a2da7527629c2"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.278718 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "c193eb76023243f589ffb5395d88ffa5"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "c95ad08251e35e48d4c012bea28d185b"
server_key_iv: "fd400ccf1a80b065800a2da7527629c2"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.289186 2824 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260810 14:38:02.308656 2845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:02.310643 2824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:38:02.310817 2824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "c193eb76023243f589ffb5395d88ffa5"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "c95ad08251e35e48d4c012bea28d185b"
server_key_iv: "fd400ccf1a80b065800a2da7527629c2"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.310973 2824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:02.348842 2824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:02.353101 2824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:02.354117 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:02.355784 2824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:02.357136 2824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:02.357265 2824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 14:38:02.357424 2824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:02.357522 2824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:02.368495 2824 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:34323
I20260810 14:38:02.369902 2824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:02.378319 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 2824
I20260810 14:38:02.378487 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:38:02.378846 21677 external_mini_cluster.cc:1428] Setting key e370faa87bc97462feea389488a73271
I20260810 14:38:02.385460 2973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:34323 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:38:02.417068 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:02.393764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:47971 (local address 127.21.43.126:38451)
0810 14:38:02.397736 (+ 3972us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:02.397742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:02.397761 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:38:02.397814 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:02.397817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:02.397882 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:02.397981 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:02.399800 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.400644 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.402419 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.402769 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.402866 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:02.411265 (+ 8399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:02.411270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:02.411273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:02.411310 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:02.413339 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:02.414191 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:02.414195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:02.414197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:02.414302 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:02.414778 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:02.414781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:02.414784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:02.415054 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:02.415191 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:02.416698 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0810 14:38:02.416847 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3869,"thread_start_us":113,"threads_started":1}
I20260810 14:38:02.418550 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:02.394357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.65:47971)
0810 14:38:02.395911 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:02.395985 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:02.397235 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:02.398362 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:02.398375 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:02.398772 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:02.399591 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:02.399615 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.400816 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.400819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:02.402264 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:02.402276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.403067 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.403071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:02.403373 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:02.404195 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:02.404218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:02.410929 (+ 6711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:02.413562 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:02.413571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:02.413593 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:02.414039 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:02.414426 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:02.414429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:02.414431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:02.414647 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:02.415257 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:02.415264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:02.415655 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 14:38:02.415888 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 14:38:02.416311 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"thread_start_us":178,"threads_started":1}
I20260810 14:38:02.420058 2974 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:02.420374 2974 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:02.421067 2974 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:02.423209 2740 ts_manager.cc:194] Registered new tserver with Master: c193eb76023243f589ffb5395d88ffa5 (127.21.43.65:34323)
I20260810 14:38:02.424922 2740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:47971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:38:02.440511 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:38:02.612710 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:02.613049 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:02.613176 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:02.618319 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:02.618429 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:02.618541 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:02.626276 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:02.627902 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:02.629078 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:02.642957 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:02.643397 2990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:02.643817 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:02.647004 2980 server_base.cc:1034] running on GCE node
I20260810 14:38:02.647606 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:02.648427 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:02.649662 2980 hybrid_clock.cc:648] HybridClock initialized: now 1786372682649627 us; error 61 us; skew 500 ppm
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:02.653167 2980 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:02.655040 2980 webserver.cc:460] Webserver started at http://127.21.43.66:38213/ using document root <none> and password file <none>
I20260810 14:38:02.655802 2980 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:02.655910 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:02.656193 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:02.675348 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "7e42c04ab47246a0bf2b1ff306a81eaa"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "b961ad4478c35c423527aae31e8074e4"
server_key_iv: "d0b59d51dd94fd2a09a6bcc37501990b"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.677387 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "7e42c04ab47246a0bf2b1ff306a81eaa"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "b961ad4478c35c423527aae31e8074e4"
server_key_iv: "d0b59d51dd94fd2a09a6bcc37501990b"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.684805 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260810 14:38:02.707695 2997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:02.709759 2980 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260810 14:38:02.709964 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "7e42c04ab47246a0bf2b1ff306a81eaa"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "b961ad4478c35c423527aae31e8074e4"
server_key_iv: "d0b59d51dd94fd2a09a6bcc37501990b"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:02.710157 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:02.742507 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:02.746475 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:02.746709 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:02.747406 2980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:02.748366 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:02.748442 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:02.748550 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:02.748595 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:02.759855 2980 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34749
I20260810 14:38:02.760847 3122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34749 every 8 connection(s)
I20260810 14:38:02.761152 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:02.763003 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 2980
I20260810 14:38:02.763198 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:38:02.763495 21677 external_mini_cluster.cc:1428] Setting key 934b876e52e976681f0d80c934aa5ece
Aug 10 14:38:02 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:38:02.790567 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:02.768999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:43473 (local address 127.21.43.126:38451)
0810 14:38:02.769189 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:02.769194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:02.771661 (+ 2467us) server_negotiation.cc:407] Connection header received
0810 14:38:02.772594 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:02.772598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:02.772668 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:02.772778 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:02.773990 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.774784 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.776765 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.777126 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.777249 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:02.782135 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:02.782140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:02.782143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:02.782180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:02.785100 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:02.786589 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:02.786594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:02.786605 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:02.786782 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:02.787678 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:02.787681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:02.787693 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:02.788208 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:02.788385 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:02.789942 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0810 14:38:02.790244 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:38:02.793807 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:02.769474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.66:43473)
0810 14:38:02.771481 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:02.771534 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:02.772370 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:02.772955 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:02.772962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:02.773288 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:02.773808 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:02.773822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.774960 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.774965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:02.776627 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:02.776646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:02.777333 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:02.777337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:02.777498 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:02.778312 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:02.778347 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:02.781754 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:02.785445 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:02.785461 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:02.785500 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:02.786336 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:02.787009 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:02.787023 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:02.787032 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:02.787419 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:02.788512 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:02.788529 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:02.789412 (+ 883us) client_negotiation.cc:769] Sending connection context
0810 14:38:02.791041 (+ 1629us) client_negotiation.cc:240] Negotiation successful
0810 14:38:02.791774 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"thread_start_us":161,"threads_started":1}
I20260810 14:38:02.795836 3123 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:02.796157 3123 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:02.796856 3123 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:02.798300 2740 ts_manager.cc:194] Registered new tserver with Master: 7e42c04ab47246a0bf2b1ff306a81eaa (127.21.43.66:34749)
I20260810 14:38:02.799126 2740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:43473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:38:02.832059 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
W20260810 14:38:03.145268 3128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:03.146904 3128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:03.147054 3128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:03.153534 3128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:03.153712 3128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:03.153855 3128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:03.162514 3128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:03.173094 3128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:03.174849 3128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:03.193676 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:03.193985 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:03.195499 3128 server_base.cc:1034] running on GCE node
W20260810 14:38:03.195889 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:03.196606 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:03.197753 3128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:03.201285 3128 hybrid_clock.cc:648] HybridClock initialized: now 1786372683201168 us; error 126 us; skew 500 ppm
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:03.206439 3128 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:03.207923 3128 webserver.cc:460] Webserver started at http://127.21.43.67:35111/ using document root <none> and password file <none>
I20260810 14:38:03.208865 3128 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:03.209028 3128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:03.209347 3128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:03.235597 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "56248736189e4b548fcb2a41c938be3d"
format_stamp: "Formatted at 2026-08-10 14:38:03 on dist-test-slave-chsk"
server_key: "fe1ce54abc46513657b6917b1e9e3082"
server_key_iv: "79a240d9057ea23c4ab53b3f6b1bb8e9"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:03.236624 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "56248736189e4b548fcb2a41c938be3d"
format_stamp: "Formatted at 2026-08-10 14:38:03 on dist-test-slave-chsk"
server_key: "fe1ce54abc46513657b6917b1e9e3082"
server_key_iv: "79a240d9057ea23c4ab53b3f6b1bb8e9"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:03.244019 3128 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260810 14:38:03.265059 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.266918 3128 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20260810 14:38:03.267066 3128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "56248736189e4b548fcb2a41c938be3d"
format_stamp: "Formatted at 2026-08-10 14:38:03 on dist-test-slave-chsk"
server_key: "fe1ce54abc46513657b6917b1e9e3082"
server_key_iv: "79a240d9057ea23c4ab53b3f6b1bb8e9"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:03.267239 3128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:03.314833 3128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:03.319060 3128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:03.319468 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:03.320495 3128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:03.321862 3128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:03.321973 3128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.322106 3128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:03.322180 3128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.349292 3128 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:33505
I20260810 14:38:03.350741 3128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:03.351253 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3128
I20260810 14:38:03.351418 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:38:03.351763 21677 external_mini_cluster.cc:1428] Setting key d436cf60966c7b1c7d9cbb5134b41aa8
I20260810 14:38:03.352062 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:33505 every 8 connection(s)
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:03.373831 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:03.356446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:46007 (local address 127.21.43.126:38451)
0810 14:38:03.360093 (+ 3647us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:03.360098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:03.360115 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:38:03.360164 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:03.360167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:03.360230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:03.360327 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:03.362519 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:03.363446 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:03.365398 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:03.365773 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:03.365854 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:03.369521 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:03.369525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:03.369528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:03.369564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:03.371287 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:03.371878 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:03.371882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:03.371884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:03.371973 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:03.372416 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:03.372419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:03.372421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:03.372690 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:03.372797 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:03.373499 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 14:38:03.373645 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3543,"thread_start_us":190,"threads_started":1}
I20260810 14:38:03.374825 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:03.355110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.67:46007)
0810 14:38:03.357994 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:03.358062 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:03.359562 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:03.360682 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:03.360693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:03.361285 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:03.362289 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:03.362315 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:03.363628 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:03.363631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:03.365228 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:03.365241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:03.366094 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:03.366098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:03.366365 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:03.367226 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:03.367251 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:03.369312 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:03.371422 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:03.371429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:03.371449 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:03.371763 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:03.372078 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:03.372081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:03.372083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:03.372300 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:03.372796 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:03.372803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:03.373111 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 14:38:03.373320 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 14:38:03.373732 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2577,"thread_start_us":263,"threads_started":1}
I20260810 14:38:03.376232 3278 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:03.376523 3278 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:03.377151 3278 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:03.378468 2740 ts_manager.cc:194] Registered new tserver with Master: 56248736189e4b548fcb2a41c938be3d (127.21.43.67:33505)
I20260810 14:38:03.379069 2740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:46007
I20260810 14:38:03.390187 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:03.390964 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 2824
I20260810 14:38:03.399870 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 2980
I20260810 14:38:03.409941 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3128
I20260810 14:38:03.418905 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 2703
I20260810 14:38:03.428054 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38451
--webserver_interface=127.21.43.126
--webserver_port=35573
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38451
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:03.547966 3283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:03.548297 3283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:03.548342 3283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:03.552635 3283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:03.552706 3283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:03.552726 3283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:03.552742 3283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:03.552791 3283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:03.558620 3283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38451
--ranger_config_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=35573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:03.559849 3283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:03.560917 3283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:03.569759 3291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:03.569823 3290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:03.569919 3283 server_base.cc:1034] running on GCE node
W20260810 14:38:03.569768 3293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:03.570595 3283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:03.571786 3283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:03.572973 3283 hybrid_clock.cc:648] HybridClock initialized: now 1786372683572958 us; error 25 us; skew 500 ppm
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:03.576382 3283 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:03.577396 3283 webserver.cc:460] Webserver started at http://127.21.43.126:35573/ using document root <none> and password file <none>
I20260810 14:38:03.577970 3283 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:03.578070 3283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:03.661248 3283 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:03.663777 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.665149 3283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:03.665324 3283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "a3dd1dd8042643a5a2e5a3e300319ce5"
format_stamp: "Formatted at 2026-08-10 14:38:01 on dist-test-slave-chsk"
server_key: "96a74ff9da2ceffcfe5ef2e9ce648629"
server_key_iv: "6a858087218ad963042d30284eeb25e1"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:03.665894 3283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:03.677472 3283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:03.680392 3283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:03.680557 3283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:03.688220 3283 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38451
I20260810 14:38:03.688257 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38451 every 8 connection(s)
I20260810 14:38:03.689358 3283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:03.694059 3362 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:38:03.695971 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Bootstrap starting.
I20260810 14:38:03.698180 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3283
I20260810 14:38:03.698697 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:34323
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:38451
--webserver_port=38913
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:03.713294 3362 log.cc:824] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:03.720966 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:03.721436 3362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Bootstrap complete.
I20260810 14:38:03.726182 3362 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:03.726665 3362 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dd1dd8042643a5a2e5a3e300319ce5, State: Initialized, Role: FOLLOWER
I20260810 14:38:03.727332 3362 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:03.727492 3362 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:03.727586 3362 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:03.727710 3362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:38:03.731003 3362 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:03.731523 3362 leader_election.cc:302] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3dd1dd8042643a5a2e5a3e300319ce5; no voters:
I20260810 14:38:03.731971 3362 leader_election.cc:288] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:38:03.732455 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:38:03.732774 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [term 2 LEADER]: Becoming Leader. State: Replica: a3dd1dd8042643a5a2e5a3e300319ce5, State: Running, Role: LEADER
I20260810 14:38:03.733242 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } }
I20260810 14:38:03.734545 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3dd1dd8042643a5a2e5a3e300319ce5. Latest consensus state: current_term: 2 leader_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } } }
I20260810 14:38:03.734764 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:03.734962 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3dd1dd8042643a5a2e5a3e300319ce5" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38451 } } }
I20260810 14:38:03.735080 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:03.735670 3362 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:03.738160 3362 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:38:03.739317 3371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:03.745954 3371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:03.746893 3371 catalog_manager.cc:1259] Loaded cluster ID: f6a8debf37f34a0da5747613ac727bf4
I20260810 14:38:03.746974 3371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:03.749972 3371 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:03.750934 3371 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Loaded TSK: 0
I20260810 14:38:03.752398 3371 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:38:03.855723 3364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:03.856137 3364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:03.856223 3364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:03.862560 3364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:03.862665 3364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:03.862757 3364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:03.871124 3364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:34323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=38913
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:03.872591 3364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:03.873992 3364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:03.886793 3409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:03.887260 3407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:03.887552 3406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:03.888873 3364 server_base.cc:1034] running on GCE node
I20260810 14:38:03.889446 3364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:03.890316 3364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:03.893653 3364 hybrid_clock.cc:648] HybridClock initialized: now 1786372683893574 us; error 111 us; skew 500 ppm
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:03.897822 3364 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:03.899168 3364 webserver.cc:460] Webserver started at http://127.21.43.65:38913/ using document root <none> and password file <none>
I20260810 14:38:03.899911 3364 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:03.900066 3364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:38:03 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:03.942314 3364 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.000s
I20260810 14:38:03.952946 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.954669 3364 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260810 14:38:03.954857 3364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "c193eb76023243f589ffb5395d88ffa5"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "c95ad08251e35e48d4c012bea28d185b"
server_key_iv: "fd400ccf1a80b065800a2da7527629c2"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:03.955715 3364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:03.986809 3364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:03.990866 3364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:03.991114 3364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:03.991969 3364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:03.993072 3364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:03.993136 3364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:03.993211 3364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:03.993233 3364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.005918 3364 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:34323
I20260810 14:38:04.006067 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:34323 every 8 connection(s)
I20260810 14:38:04.007366 3364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:04.010751 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3364
I20260810 14:38:04.011343 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:34749
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:38451
--webserver_port=38213
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:04.029650 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.009894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.65:42185)
0810 14:38:04.010475 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:04.010531 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:04.011615 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:04.012810 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:04.012819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:04.013201 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:04.013917 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.013942 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.016069 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.016073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.017585 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.017600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.019015 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.019020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.019266 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.020088 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:04.020113 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:04.022102 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:04.025209 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.025217 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.025236 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.025636 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.026159 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.026163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.026165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.026374 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.027375 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:04.027383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:04.027781 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 14:38:04.028052 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 14:38:04.028455 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":152,"threads_started":1}
I20260810 14:38:04.031863 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.009752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:42185 (local address 127.21.43.126:38451)
0810 14:38:04.011705 (+ 1953us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:04.011730 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:04.011786 (+ 56us) server_negotiation.cc:407] Connection header received
0810 14:38:04.012087 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:04.012097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:04.012407 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:04.012741 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:04.014162 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.015854 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.017762 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.018543 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.018731 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.022412 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:04.022420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:04.022441 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:04.022485 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:04.025014 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.025809 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.025817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.025823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.026024 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.026497 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.026501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.026502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.027081 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:04.027320 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:04.030432 (+ 3112us) server_negotiation.cc:299] Negotiation successful
0810 14:38:04.030796 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":126,"threads_started":1}
I20260810 14:38:04.034941 3546 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:04.035341 3546 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.036090 3546 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
W20260810 14:38:04.038964 3546 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:38451 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:38:04.039158 3546 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.039417 3546 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:04.040453 3546 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.040673 3546 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
W20260810 14:38:04.041549 3546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:38:04.183578 3551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:04.183993 3551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:04.184067 3551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:04.190609 3551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:04.190716 3551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:04.190809 3551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:04.199563 3551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=38213
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:04.201161 3551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:04.202653 3551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:04.216531 3564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:04.216754 3562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:04.217315 3551 server_base.cc:1034] running on GCE node
W20260810 14:38:04.217724 3561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:04.218364 3551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:04.219348 3551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:04.222674 3551 hybrid_clock.cc:648] HybridClock initialized: now 1786372684222622 us; error 67 us; skew 500 ppm
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:04.226912 3551 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:04.228330 3551 webserver.cc:460] Webserver started at http://127.21.43.66:38213/ using document root <none> and password file <none>
I20260810 14:38:04.229152 3551 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:04.229278 3551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:04.257083 3551 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260810 14:38:04.260702 3573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.262540 3551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:38:04.262722 3551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "7e42c04ab47246a0bf2b1ff306a81eaa"
format_stamp: "Formatted at 2026-08-10 14:38:02 on dist-test-slave-chsk"
server_key: "b961ad4478c35c423527aae31e8074e4"
server_key_iv: "d0b59d51dd94fd2a09a6bcc37501990b"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:04.263646 3551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:04.283828 3551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:04.287748 3551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:04.287976 3551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:04.288717 3551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:04.289810 3551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:04.289860 3551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.289925 3551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:04.289958 3551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.301761 3551 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34749
I20260810 14:38:04.301790 3703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34749 every 8 connection(s)
I20260810 14:38:04.303243 3551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:04.310073 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3551
I20260810 14:38:04.310583 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.43.84:60235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:33505
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:38451
--webserver_port=35111
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:04.319608 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.305616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:33059 (local address 127.21.43.126:38451)
0810 14:38:04.305788 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:04.305793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:04.306837 (+ 1044us) server_negotiation.cc:407] Connection header received
0810 14:38:04.307993 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:04.307997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:04.308061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:04.308168 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:04.309377 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.310026 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.312387 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.312729 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.312862 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.315639 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:04.315652 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:04.315655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:04.315688 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:04.317258 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.317859 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.317863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.317865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.317971 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.318352 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.318356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.318357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.318569 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:04.318688 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:04.319268 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 14:38:04.319416 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:38:04.320776 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.306127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.66:33059)
0810 14:38:04.306662 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:04.306721 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:04.307758 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:04.308374 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:04.308382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:04.308685 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:04.309213 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.309228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.310212 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.310217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.312224 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.312240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.312920 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.312924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.313108 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.313776 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:04.313795 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:04.315433 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:04.317407 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.317414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.317440 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.317741 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.318067 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.318070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.318072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.318243 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.318684 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:04.318691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:04.318979 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:38:04.319213 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 14:38:04.319568 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":158,"threads_started":1}
I20260810 14:38:04.322119 3704 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:04.322393 3704 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.322940 3704 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
W20260810 14:38:04.324599 3704 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:38451 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:38:04.324749 3704 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.324961 3704 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:04.326758 3704 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.326972 3704 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
W20260810 14:38:04.327819 3704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:38:04.470714 3708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:04.471199 3708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:04.471302 3708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:04.476464 3708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:04.476573 3708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:04.476686 3708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:04.483938 3708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.43.84:60235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:33505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=35111
--tserver_master_addrs=127.21.43.126:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:04.485679 3708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:04.486964 3708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:04.499418 3715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:04.500154 3714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:04.500452 3717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:04.502244 3708 server_base.cc:1034] running on GCE node
I20260810 14:38:04.502934 3708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:04.503937 3708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:04.505156 3708 hybrid_clock.cc:648] HybridClock initialized: now 1786372684505129 us; error 47 us; skew 500 ppm
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:04.508749 3708 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:04.509821 3708 webserver.cc:460] Webserver started at http://127.21.43.67:35111/ using document root <none> and password file <none>
I20260810 14:38:04.510440 3708 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:04.510560 3708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:04.539052 3708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:38:04.542037 3725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.543645 3708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:04.543819 3708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "56248736189e4b548fcb2a41c938be3d"
format_stamp: "Formatted at 2026-08-10 14:38:03 on dist-test-slave-chsk"
server_key: "fe1ce54abc46513657b6917b1e9e3082"
server_key_iv: "79a240d9057ea23c4ab53b3f6b1bb8e9"
server_key_version: "kuduclusterkey@0"
I20260810 14:38:04.544422 3708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:04.563989 3708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:04.567757 3708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:04.567974 3708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:04.568737 3708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:04.569864 3708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:04.569931 3708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.569998 3708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:04.570029 3708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:04.582319 3708 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:33505
I20260810 14:38:04.582381 3853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:33505 every 8 connection(s)
I20260810 14:38:04.583544 3708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:04.591745 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3708
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:04.599488 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.585561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:35357 (local address 127.21.43.126:38451)
0810 14:38:04.585744 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:04.585749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:04.586762 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 14:38:04.587824 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:04.587828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:04.587888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:04.588009 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:04.589295 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.590173 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.591476 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.591835 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.591950 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.594886 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:04.594890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:04.594893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:04.594922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:04.596704 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.597397 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.597400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.597402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.597500 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.597945 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.597949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.597950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.598192 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:04.598302 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:04.599065 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 14:38:04.599271 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:38:04.600494 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.585949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38451 (local address 127.21.43.67:35357)
0810 14:38:04.586565 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:04.586634 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:04.587591 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:04.588171 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:04.588178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:04.588459 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:04.589124 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.589142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.590312 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.590315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.591350 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:04.591360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.592035 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.592040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:04.592256 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.592982 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:04.593008 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:04.594668 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:04.596855 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.596862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.596881 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.597264 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.597627 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:04.597632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:04.597635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:04.597831 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:04.598368 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:04.598376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:04.598741 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 14:38:04.598982 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 14:38:04.599393 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":155,"threads_started":1}
I20260810 14:38:04.601787 3854 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38451
I20260810 14:38:04.602082 3854 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:04.602700 3854 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:04.602794 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:04.592541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.21.43.126:38451)
0810 14:38:04.592912 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:04.592920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:04.593394 (+ 474us) server_negotiation.cc:407] Connection header received
0810 14:38:04.593514 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:04.593517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:04.593583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:04.593681 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:04.595179 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.595877 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.596799 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:04.597061 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:04.597172 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:04.598742 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:04.598745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:04.598748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:04.598778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:04.600550 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.601284 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.601286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.601287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.601387 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:04.601708 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:04.601711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:04.601712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:04.601959 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:04.602184 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:04.602432 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 14:38:04.602576 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":140,"threads_started":1}
I20260810 14:38:04.604647 3323 ts_manager.cc:194] Registered new tserver with Master: 56248736189e4b548fcb2a41c938be3d (127.21.43.67:33505)
I20260810 14:38:04.607689 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:35357
Aug 10 14:38:04 dist-test-slave-chsk krb5kdc[2197](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:05.041819 3546 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:05.042198 3546 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:05.043260 3323 ts_manager.cc:194] Registered new tserver with Master: c193eb76023243f589ffb5395d88ffa5 (127.21.43.65:34323)
I20260810 14:38:05.044121 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:42185
I20260810 14:38:05.328042 3704 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:05.328385 3704 heartbeater.cc:511] Master 127.21.43.126:38451 requested a full tablet report, sending...
I20260810 14:38:05.329102 3323 ts_manager.cc:194] Registered new tserver with Master: 7e42c04ab47246a0bf2b1ff306a81eaa (127.21.43.66:34749)
I20260810 14:38:05.329720 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:33059
I20260810 14:38:05.340215 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:05.351998 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.341786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.21.43.126:38451)
0810 14:38:05.342046 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.342051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.342337 (+ 286us) server_negotiation.cc:407] Connection header received
0810 14:38:05.342384 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.342386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.342445 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.342529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:05.343442 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.344104 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.345203 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.345525 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.345629 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.347555 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:05.347558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:05.347561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:05.347591 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:05.349414 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:05.349931 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:05.349935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:05.349937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:05.350033 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:05.350581 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:05.350585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:05.350587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:05.350843 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:05.350994 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.351682 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.351811 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260810 14:38:05.356544 3323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:38:05.361225 3323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 14:38:05 dist-test-slave-chsk krb5kdc[2197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for HTTP/127.21.43.84@KRBTEST.COM
I20260810 14:38:05.382371 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.373067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34749 (local address 127.0.0.1:54152)
0810 14:38:05.375210 (+ 2143us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.375276 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:05.375426 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:05.376126 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:05.376132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:05.376162 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:05.376329 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.376339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.378219 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.378222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.380509 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.380519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.381780 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.381784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.381894 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.381932 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:38:05.382056 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 14:38:05.382204 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"thread_start_us":131,"threads_started":1}
I20260810 14:38:05.383807 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.374442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.21.43.66:34749)
0810 14:38:05.375058 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.375066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.375560 (+ 494us) server_negotiation.cc:407] Connection header received
0810 14:38:05.375669 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.375674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.375926 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.376083 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:05.376471 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.378095 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.380653 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.381458 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.381564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.381716 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.383549 (+ 1833us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.383660 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":139,"threads_started":1}
I20260810 14:38:05.385528 3879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.373484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34323 (local address 127.0.0.1:51456)
0810 14:38:05.376216 (+ 2732us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.376230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:05.377115 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:05.377810 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:05.377812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:05.377828 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:05.377921 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.377927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.380132 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.380135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.383786 (+ 3651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.383795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.385130 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.385133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.385231 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.385255 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:38:05.385324 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:38:05.385396 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":98,"threads_started":1}
I20260810 14:38:05.386063 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.373980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.21.43.65:34323)
0810 14:38:05.375426 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.375433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.376319 (+ 886us) server_negotiation.cc:407] Connection header received
0810 14:38:05.377286 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.377292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.377494 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.377623 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:05.378806 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.379990 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.383947 (+ 3957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.384804 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.384886 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.385023 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.385818 (+ 795us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.385926 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1323,"thread_start_us":83,"threads_started":1}
I20260810 14:38:05.387709 3629 tablet_service.cc:1467] Processing CreateTablet for tablet c5e627152ad2464da0c794348f23a8db (DEFAULT_TABLE table=test-table [id=ae5894d8a6894b1696294b01e3880384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:05.388015 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.373967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:33505 (local address 127.0.0.1:45776)
0810 14:38:05.376431 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.376444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:05.376534 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:05.378552 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:05.378555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:05.378572 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:05.378661 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.378667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.382183 (+ 3516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.382186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.383287 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.383297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.387614 (+ 4317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.387618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.387711 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.387744 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:38:05.387821 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:38:05.387894 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2393,"thread_start_us":94,"threads_started":1}
I20260810 14:38:05.388969 3629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e627152ad2464da0c794348f23a8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:05.389762 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.374146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.21.43.67:33505)
0810 14:38:05.376130 (+ 1984us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.376136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.377027 (+ 891us) server_negotiation.cc:407] Connection header received
0810 14:38:05.378095 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.378102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.378288 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.378417 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:05.380982 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.382061 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.386449 (+ 4388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.387289 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.387398 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.387534 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.389498 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.389621 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1864,"thread_start_us":133,"threads_started":1}
I20260810 14:38:05.391255 3470 tablet_service.cc:1467] Processing CreateTablet for tablet c5e627152ad2464da0c794348f23a8db (DEFAULT_TABLE table=test-table [id=ae5894d8a6894b1696294b01e3880384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:05.392460 3470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e627152ad2464da0c794348f23a8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:05.392643 3781 tablet_service.cc:1467] Processing CreateTablet for tablet c5e627152ad2464da0c794348f23a8db (DEFAULT_TABLE table=test-table [id=ae5894d8a6894b1696294b01e3880384]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:05.393623 3781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e627152ad2464da0c794348f23a8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:05.411751 3884 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Bootstrap starting.
I20260810 14:38:05.414858 3884 tablet_bootstrap.cc:654] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:05.415974 3884 log.cc:824] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:05.419785 3883 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Bootstrap starting.
I20260810 14:38:05.420210 3884 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: No bootstrap required, opened a new log
I20260810 14:38:05.420456 3884 ts_tablet_manager.cc:1397] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 14:38:05.424916 3884 raft_consensus.cc:348] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.425338 3884 raft_consensus.cc:374] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:05.426370 3884 raft_consensus.cc:729] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e42c04ab47246a0bf2b1ff306a81eaa, State: Initialized, Role: FOLLOWER
I20260810 14:38:05.426990 3884 consensus_queue.cc:260] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.427740 3882 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Bootstrap starting.
I20260810 14:38:05.427188 3883 tablet_bootstrap.cc:654] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:05.428422 3884 ts_tablet_manager.cc:1428] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260810 14:38:05.428920 3704 heartbeater.cc:503] Master 127.21.43.126:38451 was elected leader, sending a full tablet report...
I20260810 14:38:05.428897 3883 log.cc:824] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:05.437255 3882 tablet_bootstrap.cc:654] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:05.438306 3882 log.cc:824] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:05.446378 3882 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: No bootstrap required, opened a new log
I20260810 14:38:05.447885 3883 tablet_bootstrap.cc:492] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: No bootstrap required, opened a new log
I20260810 14:38:05.448230 3883 ts_tablet_manager.cc:1397] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20260810 14:38:05.453766 3883 raft_consensus.cc:348] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.454445 3882 ts_tablet_manager.cc:1397] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20260810 14:38:05.458673 3882 raft_consensus.cc:348] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.459108 3882 raft_consensus.cc:374] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:05.459236 3882 raft_consensus.cc:729] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193eb76023243f589ffb5395d88ffa5, State: Initialized, Role: FOLLOWER
I20260810 14:38:05.459240 3883 raft_consensus.cc:374] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:05.459424 3883 raft_consensus.cc:729] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56248736189e4b548fcb2a41c938be3d, State: Initialized, Role: FOLLOWER
I20260810 14:38:05.459873 3882 consensus_queue.cc:260] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.460028 3883 consensus_queue.cc:260] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.461231 3882 ts_tablet_manager.cc:1428] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:38:05.461231 3883 ts_tablet_manager.cc:1428] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260810 14:38:05.461678 3546 heartbeater.cc:503] Master 127.21.43.126:38451 was elected leader, sending a full tablet report...
I20260810 14:38:05.464653 3854 heartbeater.cc:503] Master 127.21.43.126:38451 was elected leader, sending a full tablet report...
I20260810 14:38:05.481309 3890 raft_consensus.cc:482] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:05.481556 3890 raft_consensus.cc:504] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.482998 3890 leader_election.cc:288] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e42c04ab47246a0bf2b1ff306a81eaa (127.21.43.66:34749), 56248736189e4b548fcb2a41c938be3d (127.21.43.67:33505)
I20260810 14:38:05.490420 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.484546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:49997 (local address 127.21.43.66:34749)
0810 14:38:05.484724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.484729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.486046 (+ 1317us) server_negotiation.cc:407] Connection header received
0810 14:38:05.486259 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.486263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.486312 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.486358 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:05.486785 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.487589 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.489188 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.489703 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.489798 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.489851 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.490203 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.490283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 14:38:05.491088 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.483604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34749 (local address 127.21.43.65:49997)
0810 14:38:05.485948 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.485964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:05.486136 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:05.486490 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:05.486494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:05.486509 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:05.486660 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.486666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.487755 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.487758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.489035 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.489048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.489951 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.489954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.490062 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.490084 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:38:05.490848 (+ 764us) client_negotiation.cc:240] Negotiation successful
0810 14:38:05.490927 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2266,"thread_start_us":155,"threads_started":1}
I20260810 14:38:05.491950 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e627152ad2464da0c794348f23a8db" candidate_uuid: "c193eb76023243f589ffb5395d88ffa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" is_pre_election: true
I20260810 14:38:05.492302 3633 raft_consensus.cc:2454] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193eb76023243f589ffb5395d88ffa5 in term 0.
I20260810 14:38:05.492941 3421 leader_election.cc:302] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e42c04ab47246a0bf2b1ff306a81eaa, c193eb76023243f589ffb5395d88ffa5; no voters:
I20260810 14:38:05.493261 3890 raft_consensus.cc:2781] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:05.493330 3890 raft_consensus.cc:482] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:05.493367 3890 raft_consensus.cc:3037] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:05.493374 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.484837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:49145 (local address 127.21.43.67:33505)
0810 14:38:05.484964 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.484969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.488998 (+ 4029us) server_negotiation.cc:407] Connection header received
0810 14:38:05.489215 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.489218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.489275 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.489324 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:05.489701 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.490458 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.492002 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.492624 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.492718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.492766 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.493171 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.493252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:38:05.493749 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.484057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:33505 (local address 127.21.43.65:49145)
0810 14:38:05.488895 (+ 4838us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.488910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:05.489073 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:05.489454 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:05.489457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:05.489473 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:05.489578 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.489583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.490607 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.490610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.491848 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:05.491858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.492861 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.492864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:05.492967 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.492988 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:38:05.493526 (+ 538us) client_negotiation.cc:240] Negotiation successful
0810 14:38:05.493608 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4765,"thread_start_us":103,"threads_started":1}
I20260810 14:38:05.494650 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e627152ad2464da0c794348f23a8db" candidate_uuid: "c193eb76023243f589ffb5395d88ffa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56248736189e4b548fcb2a41c938be3d" is_pre_election: true
I20260810 14:38:05.494879 3890 raft_consensus.cc:504] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.495018 3805 raft_consensus.cc:2454] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193eb76023243f589ffb5395d88ffa5 in term 0.
I20260810 14:38:05.495337 3890 leader_election.cc:288] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [CANDIDATE]: Term 1 election: Requested vote from peers 7e42c04ab47246a0bf2b1ff306a81eaa (127.21.43.66:34749), 56248736189e4b548fcb2a41c938be3d (127.21.43.67:33505)
I20260810 14:38:05.495847 3805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e627152ad2464da0c794348f23a8db" candidate_uuid: "c193eb76023243f589ffb5395d88ffa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56248736189e4b548fcb2a41c938be3d"
I20260810 14:38:05.495988 3805 raft_consensus.cc:3037] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:05.496443 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e627152ad2464da0c794348f23a8db" candidate_uuid: "c193eb76023243f589ffb5395d88ffa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa"
I20260810 14:38:05.496593 3633 raft_consensus.cc:3037] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:05.497429 3805 raft_consensus.cc:2454] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193eb76023243f589ffb5395d88ffa5 in term 1.
I20260810 14:38:05.498011 3633 raft_consensus.cc:2454] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193eb76023243f589ffb5395d88ffa5 in term 1.
I20260810 14:38:05.498010 3421 leader_election.cc:302] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56248736189e4b548fcb2a41c938be3d, c193eb76023243f589ffb5395d88ffa5; no voters:
I20260810 14:38:05.498219 3890 raft_consensus.cc:2781] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:05.498490 3890 raft_consensus.cc:686] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [term 1 LEADER]: Becoming Leader. State: Replica: c193eb76023243f589ffb5395d88ffa5, State: Running, Role: LEADER
I20260810 14:38:05.498888 3890 consensus_queue.cc:237] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } }
I20260810 14:38:05.502880 3323 catalog_manager.cc:5495] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c193eb76023243f589ffb5395d88ffa5 (127.21.43.65). New cstate: current_term: 1 leader_uuid: "c193eb76023243f589ffb5395d88ffa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c193eb76023243f589ffb5395d88ffa5" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34323 } health_report { overall_health: HEALTHY } } }
W20260810 14:38:05.510383 3548 tablet.cc:2369] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:38:05.520524 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.515200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.21.43.65:34323)
0810 14:38:05.515380 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:05.515385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:05.515399 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:38:05.515489 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:05.515492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:05.515559 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:05.515679 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:05.516021 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.516839 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.518145 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:05.518493 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:05.518603 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:05.519054 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:05.519680 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:05.519799 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:05.520265 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 14:38:05.520335 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 14:38:05.529510 3805 raft_consensus.cc:1261] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 1 FOLLOWER]: Refusing update from remote peer c193eb76023243f589ffb5395d88ffa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:05.530808 3633 raft_consensus.cc:1261] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 1 FOLLOWER]: Refusing update from remote peer c193eb76023243f589ffb5395d88ffa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:05.531905 3890 consensus_queue.cc:1035] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e42c04ab47246a0bf2b1ff306a81eaa" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:05.532227 3894 consensus_queue.cc:1035] T c5e627152ad2464da0c794348f23a8db P c193eb76023243f589ffb5395d88ffa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56248736189e4b548fcb2a41c938be3d" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:38:05.556337 3705 tablet.cc:2369] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:38:05.561828 3897 mvcc.cc:205] Tried to move back new op lower bound from 7316982519916331008 to 7316982519806967808. Current Snapshot: MvccSnapshot[applied={T|T < 7316982519916331008}]
I20260810 14:38:05.568261 3900 mvcc.cc:205] Tried to move back new op lower bound from 7316982519916331008 to 7316982519806967808. Current Snapshot: MvccSnapshot[applied={T|T < 7316982519916331008}]
I20260810 14:38:05.568296 3898 mvcc.cc:205] Tried to move back new op lower bound from 7316982519916331008 to 7316982519806967808. Current Snapshot: MvccSnapshot[applied={T|T < 7316982519916331008}]
I20260810 14:38:05.580437 3323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:05.580673 3323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:05.583802 3323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3dd1dd8042643a5a2e5a3e300319ce5: Sending DeleteTablet for 3 replicas of tablet c5e627152ad2464da0c794348f23a8db
I20260810 14:38:05.584817 3629 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e627152ad2464da0c794348f23a8db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:05 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:54152
I20260810 14:38:05.584966 3781 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e627152ad2464da0c794348f23a8db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:05 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:45776
I20260810 14:38:05.585479 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e627152ad2464da0c794348f23a8db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:05 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:51456
I20260810 14:38:05.586410 3906 tablet_replica.cc:330] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: stopping tablet replica
I20260810 14:38:05.586740 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3364
I20260810 14:38:05.586863 3906 raft_consensus.cc:2229] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:05.590152 3906 raft_consensus.cc:2258] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:05.592272 3906 ts_tablet_manager.cc:1905] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:05.593284 3907 tablet_replica.cc:330] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: stopping tablet replica
I20260810 14:38:05.593617 3907 raft_consensus.cc:2229] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:05.593957 3907 raft_consensus.cc:2258] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:05.595950 3906 ts_tablet_manager.cc:1918] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:05.596128 3906 log.cc:1196] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/c5e627152ad2464da0c794348f23a8db
I20260810 14:38:05.596578 3906 ts_tablet_manager.cc:1939] T c5e627152ad2464da0c794348f23a8db P 7e42c04ab47246a0bf2b1ff306a81eaa: Deleting consensus metadata
I20260810 14:38:05.598165 3310 catalog_manager.cc:4841] TS 7e42c04ab47246a0bf2b1ff306a81eaa (127.21.43.66:34749): tablet c5e627152ad2464da0c794348f23a8db (table test-table [id=ae5894d8a6894b1696294b01e3880384]) successfully deleted
I20260810 14:38:05.602977 3907 ts_tablet_manager.cc:1905] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:05.606231 3907 ts_tablet_manager.cc:1918] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:05.606366 3907 log.cc:1196] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/c5e627152ad2464da0c794348f23a8db
I20260810 14:38:05.606722 3907 ts_tablet_manager.cc:1939] T c5e627152ad2464da0c794348f23a8db P 56248736189e4b548fcb2a41c938be3d: Deleting consensus metadata
W20260810 14:38:05.607033 3309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:34323 (error 108)
I20260810 14:38:05.608112 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3551
I20260810 14:38:05.608263 3310 catalog_manager.cc:4841] TS 56248736189e4b548fcb2a41c938be3d (127.21.43.67:33505): tablet c5e627152ad2464da0c794348f23a8db (table test-table [id=ae5894d8a6894b1696294b01e3880384]) successfully deleted
I20260810 14:38:05.613847 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:05.613392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34323 (local address 127.0.0.1:51472)
0810 14:38:05.613641 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:05.613734 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260810 14:38:05.614087 3309 catalog_manager.cc:4584] TS c193eb76023243f589ffb5395d88ffa5 (127.21.43.65:34323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5e627152ad2464da0c794348f23a8db: Network error: Client connection negotiation failed: client connection to 127.21.43.65:34323: connect: Connection refused (error 111)
I20260810 14:38:05.621248 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3708
I20260810 14:38:05.632508 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3283
I20260810 14:38:05.644018 21677 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:38:05.990051 21677 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:38:05.990193 21677 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:38:06.346053 21677 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:38:06.346 UTC [2231] LOG: received smart shutdown request
2026-08-10 14:38:06.351 UTC [2231] LOG: background worker "logical replication launcher" (PID 2242) exited with exit code 1
2026-08-10 14:38:06.351 UTC [2235] LOG: shutting down
2026-08-10 14:38:06.554 UTC [2231] LOG: database system is shut down
2026-08-10T14:38:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50989 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:38:06 dist-test-slave-chsk krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:38:06 dist-test-slave-chsk krb5kdc[3919](info): set up 2 sockets
Aug 10 14:38:06 dist-test-slave-chsk krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260810 14:38:07.804518 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:38:07 dist-test-slave-chsk krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:38:07.950695 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34051
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:08.071913 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:08.072321 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:08.072405 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:08.077023 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:08.077178 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:08.077220 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:08.077262 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:08.077304 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:08.083421 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:08.084722 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:08.085932 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:08.095944 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:08.095944 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:08.096009 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:08.096417 3932 server_base.cc:1034] running on GCE node
I20260810 14:38:08.097106 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:08.098368 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:08.099604 3932 hybrid_clock.cc:648] HybridClock initialized: now 1786372688099595 us; error 35 us; skew 500 ppm
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:08.102864 3932 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:08.104612 3932 webserver.cc:460] Webserver started at http://127.21.43.126:39801/ using document root <none> and password file <none>
I20260810 14:38:08.105451 3932 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:08.105567 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:08.105877 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:08.108712 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "bf1818ef1a594348aea5e7779870912a"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "b0cf6649ac5ef21cecad281c0663917d"
server_key_iv: "a289da4fe7323c19c73afad9475cbdfb"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.109458 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "bf1818ef1a594348aea5e7779870912a"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "b0cf6649ac5ef21cecad281c0663917d"
server_key_iv: "a289da4fe7323c19c73afad9475cbdfb"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.114789 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 14:38:08.118382 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:08.120060 3932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:08.120256 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "bf1818ef1a594348aea5e7779870912a"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "b0cf6649ac5ef21cecad281c0663917d"
server_key_iv: "a289da4fe7323c19c73afad9475cbdfb"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.120417 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:08.140228 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:08.143950 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:08.144191 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:08.152395 3932 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:34051
I20260810 14:38:08.152423 4008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:34051 every 8 connection(s)
I20260810 14:38:08.153502 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:08.156561 4009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:08.159407 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 3932
I20260810 14:38:08.159578 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:38:08.159894 21677 external_mini_cluster.cc:1428] Setting key 9ae54c638674d836c68702362c49bb57
I20260810 14:38:08.161605 4009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.125@KRBTEST.COM" created.
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:08.179059 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.164539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34051 (local address 127.0.0.1:55916)
0810 14:38:08.165187 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.165231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.166185 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.166905 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.166911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.167442 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.167719 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.167735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.168940 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.168944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.169795 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.169805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.170492 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.170496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.170626 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.171279 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.171297 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.173343 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.175554 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.175567 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.175582 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.175946 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.176322 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.176328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.176333 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.176501 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.177178 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.177188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.177421 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.177671 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.178215 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":105,"spinlock_wait_cycles":41728,"thread_start_us":181,"threads_started":1}
I20260810 14:38:08.179055 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.164539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.21.43.126:34051)
0810 14:38:08.165034 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.165054 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.165350 (+ 296us) server_negotiation.cc:407] Connection header received
0810 14:38:08.166514 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.166519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.166706 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.166905 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.167883 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.168808 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.169920 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.170318 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.170500 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.173534 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.173540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.173553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.173579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.175425 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.176075 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.176092 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.176094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.176217 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.176608 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.176610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.176611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.177034 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.177155 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.177827 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.178056 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":174,"threads_started":1}
I20260810 14:38:08.181916 4009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:38:08.182950 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.182637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45104)
0810 14:38:08.182797 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.182876 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 14:38:08.183282 3950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
I20260810 14:38:08.184264 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.184008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45114)
0810 14:38:08.184151 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.184203 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 14:38:08.184437 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37629: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
I20260810 14:38:08.191336 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:37629
--webserver_interface=127.21.43.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:08.233732 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } attempt: 1
I20260810 14:38:08.234768 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.234380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45130)
0810 14:38:08.234586 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.234668 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 14:38:08.235671 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.235405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45132)
0810 14:38:08.235530 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.235598 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:38:08.235839 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37629: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
I20260810 14:38:08.304065 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } attempt: 2
I20260810 14:38:08.305052 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.304670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45142)
0810 14:38:08.304882 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.304959 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 14:38:08.305895 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.305633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45148)
0810 14:38:08.305763 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.305821 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 14:38:08.306069 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37629: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
W20260810 14:38:08.314658 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:08.314991 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:08.315034 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:08.319438 4016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:08.319531 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:08.319552 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:08.319571 4016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:08.319590 4016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:08.325727 4016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.43.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:08.327008 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:08.328215 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:08.337797 4025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:08.337798 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:08.337944 4016 server_base.cc:1034] running on GCE node
W20260810 14:38:08.337798 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:08.338824 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:08.340160 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:08.341361 4016 hybrid_clock.cc:648] HybridClock initialized: now 1786372688341347 us; error 29 us; skew 500 ppm
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:08.344652 4016 init.cc:377] Logged in from keytab as kudu/127.21.43.125@KRBTEST.COM (short username kudu)
I20260810 14:38:08.345709 4016 webserver.cc:460] Webserver started at http://127.21.43.125:33937/ using document root <none> and password file <none>
I20260810 14:38:08.346331 4016 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:08.346397 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:08.346669 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:08.348973 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data/instance:
uuid: "a1754f188183443caff9a2711e64b7ce"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "ca7e074b7ae7d025360cb33208ffad14"
server_key_iv: "0f7fc608ce4c3d7a78979fd0b68708f4"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.349624 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal/instance:
uuid: "a1754f188183443caff9a2711e64b7ce"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "ca7e074b7ae7d025360cb33208ffad14"
server_key_iv: "0f7fc608ce4c3d7a78979fd0b68708f4"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.353850 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 14:38:08.356824 4033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:08.358245 4016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:38:08.358381 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "a1754f188183443caff9a2711e64b7ce"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "ca7e074b7ae7d025360cb33208ffad14"
server_key_iv: "0f7fc608ce4c3d7a78979fd0b68708f4"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.358490 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:08.394739 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:08.397257 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } attempt: 3
I20260810 14:38:08.398108 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.397819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45160)
0810 14:38:08.397963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.398031 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 14:38:08.398597 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:08.398819 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.398577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45168)
0810 14:38:08.398688 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.398745 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:38:08.398814 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260810 14:38:08.398991 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37629: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37629: connect: Connection refused (error 111)
I20260810 14:38:08.406337 4016 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.125:37629
I20260810 14:38:08.406409 4088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.125:37629 every 8 connection(s)
I20260810 14:38:08.407523 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
I20260810 14:38:08.410231 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4016
I20260810 14:38:08.410379 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-1/wal/instance
I20260810 14:38:08.410692 21677 external_mini_cluster.cc:1428] Setting key e0542d6150cdfa0f1c26991822d5873e
I20260810 14:38:08.410883 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:08.415397 4089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.124@KRBTEST.COM" created.
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:08.431527 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.417522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.21.43.126:34051)
0810 14:38:08.417710 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.417715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.418566 (+ 851us) server_negotiation.cc:407] Connection header received
0810 14:38:08.419656 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.419660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.419730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.419845 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.421010 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.421557 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.423062 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.423366 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.423458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.426737 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.426744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.426750 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.426788 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.428862 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.429516 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.429523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.429526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.429648 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.430110 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.430117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.430120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.430394 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.430518 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.431135 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.431316 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:38:08.432466 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.417871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34051 (local address 127.0.0.1:55926)
0810 14:38:08.418389 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.418441 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.419441 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.419990 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.419997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.420325 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.420862 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.420875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.421760 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.421763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.422917 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.422932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.423537 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.423544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.423833 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.424653 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.424683 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.426536 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.429049 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.429055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.429072 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.429388 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.429783 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.429787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.429788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.429971 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.430528 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.430535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.430840 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.431050 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.431447 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":170,"threads_started":1}
I20260810 14:38:08.434087 4089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
I20260810 14:38:08.441753 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:35393
--webserver_interface=127.21.43.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:08.444317 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.434699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45176)
0810 14:38:08.434839 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.434871 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.434974 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.435735 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.435738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.435886 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.435982 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.435987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.437109 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.437115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.437825 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.437839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.438530 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.438534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.438613 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.439127 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.439180 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.440525 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.442692 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.442696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.442698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.442895 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.443237 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.443240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.443242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.443326 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.443792 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.443795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.443879 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.443997 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.444136 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 14:38:08.444388 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.434814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.21.43.125:37629)
0810 14:38:08.435306 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.435316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.435337 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:38:08.435374 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.435378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.435527 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.435645 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.436109 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.436928 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.437957 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.438409 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.438497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.440678 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.440692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.440712 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.440734 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.442570 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.443010 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.443016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.443020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.443105 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.443418 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.443423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.443425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.443685 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.443830 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.444112 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.444246 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":209,"threads_started":1}
I20260810 14:38:08.446426 4089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:38:08.447314 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.446977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53312)
0810 14:38:08.447112 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.447241 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:38:08.447605 4035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
I20260810 14:38:08.448642 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.448380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53322)
0810 14:38:08.448536 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.448584 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:38:08.448899 4089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:35393: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
I20260810 14:38:08.498205 4089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } attempt: 1
I20260810 14:38:08.499271 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.498891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53338)
0810 14:38:08.499061 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.499127 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:38:08.500005 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.499788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53346)
0810 14:38:08.499900 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.499954 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:38:08.500125 4089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:35393: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
I20260810 14:38:08.542189 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } attempt: 4
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:38:08.552604 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.542851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45186)
0810 14:38:08.543073 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.543091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.543247 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.543527 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.543530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.543748 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.543862 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.543868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.545300 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.545307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.546437 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.546448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.546960 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.546964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.547079 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.547710 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.547723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.549019 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.550979 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.550982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.550983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.551284 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.551630 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.551633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.551635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.551723 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.552177 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.552179 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.552270 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.552351 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.552461 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 14:38:08.552752 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.542957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.21.43.125:37629)
0810 14:38:08.543092 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.543108 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.543166 (+ 58us) server_negotiation.cc:407] Connection header received
0810 14:38:08.543362 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.543371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.543419 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.543510 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.543995 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.545175 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.546573 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.546833 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.546951 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.549185 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.549189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.549193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.549226 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.550846 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.551390 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.551394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.551395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.551489 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.551839 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.551842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.551844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.552074 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.552186 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.552453 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.552588 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:38:08.553483 4009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:38:08.554231 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.553936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53348)
0810 14:38:08.554071 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.554130 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 14:38:08.554929 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.554708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53360)
0810 14:38:08.554814 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.554868 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 14:38:08.555068 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:35393: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
W20260810 14:38:08.563815 4096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:08.564111 4096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:08.564220 4096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:08.568333 4096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260810 14:38:08.568302 4089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } attempt: 2
W20260810 14:38:08.568415 4096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:08.568454 4096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:08.568513 4096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:08.568568 4096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:08.569162 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.568827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53372)
0810 14:38:08.569000 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.569070 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:38:08.569932 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.569696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53386)
0810 14:38:08.569815 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.569869 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:38:08.570072 4089 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:35393: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
I20260810 14:38:08.574374 4096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34051,127.21.43.125:37629,127.21.43.124:35393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.43.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:08.575515 4096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:08.576560 4096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:08.585322 4106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:08.585335 4103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:08.585592 4096 server_base.cc:1034] running on GCE node
W20260810 14:38:08.585322 4104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:08.586230 4096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:08.587479 4096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:08.588689 4096 hybrid_clock.cc:648] HybridClock initialized: now 1786372688588659 us; error 48 us; skew 500 ppm
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:08.591889 4096 init.cc:377] Logged in from keytab as kudu/127.21.43.124@KRBTEST.COM (short username kudu)
I20260810 14:38:08.592850 4096 webserver.cc:460] Webserver started at http://127.21.43.124:40669/ using document root <none> and password file <none>
I20260810 14:38:08.593468 4096 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:08.593571 4096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:08.593813 4096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:08.596117 4096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data/instance:
uuid: "32da32697a494304a91e9f4c280d3680"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "eccc50543a1736dc3d3288090c54d0e5"
server_key_iv: "b5f54541968dd6e81898efd6b8cfcd62"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.596707 4096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal/instance:
uuid: "32da32697a494304a91e9f4c280d3680"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "eccc50543a1736dc3d3288090c54d0e5"
server_key_iv: "b5f54541968dd6e81898efd6b8cfcd62"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.600625 4096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:38:08.603122 4113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:08.604513 4096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:38:08.604705 4096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "32da32697a494304a91e9f4c280d3680"
format_stamp: "Formatted at 2026-08-10 14:38:08 on dist-test-slave-chsk"
server_key: "eccc50543a1736dc3d3288090c54d0e5"
server_key_iv: "b5f54541968dd6e81898efd6b8cfcd62"
server_key_version: "encryptionkey@0"
I20260810 14:38:08.604837 4096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:08.614238 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } attempt: 1
I20260810 14:38:08.615298 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.614872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53402)
0810 14:38:08.615072 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.615190 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 14:38:08.616087 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.615855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53412)
0810 14:38:08.615967 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.616032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:38:08.616228 4009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:35393: Network error: Client connection negotiation failed: client connection to 127.21.43.124:35393: connect: Connection refused (error 111)
I20260810 14:38:08.621325 4096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:08.625291 4096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:08.625530 4096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:08.633234 4096 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.124:35393
I20260810 14:38:08.633296 4173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.124:35393 every 8 connection(s)
I20260810 14:38:08.634327 4096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
I20260810 14:38:08.637533 4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:08.640503 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4096
I20260810 14:38:08.640650 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372517778458-21677-0/minicluster-data/master-2/wal/instance
I20260810 14:38:08.640955 21677 external_mini_cluster.cc:1428] Setting key c6e67a7e103d1cf61718a223267efacf
I20260810 14:38:08.642297 4174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:38:08.655320 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.644360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.21.43.126:34051)
0810 14:38:08.644523 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.644528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.644914 (+ 386us) server_negotiation.cc:407] Connection header received
0810 14:38:08.645118 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.645121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.645174 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.645264 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.646153 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.646687 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.647861 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.648076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.648195 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.650932 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.650939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.650946 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.650985 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.652718 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.653306 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.653309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.653310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.653392 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.653803 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.653807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.653809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.654130 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.654212 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.654937 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.655107 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:38:08.656378 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.642528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.21.43.124:35393)
0810 14:38:08.643022 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.643037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.643079 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:38:08.643810 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.643828 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.644226 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.644590 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.645405 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.646660 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.647584 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.648075 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.648248 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.651448 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.651456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.651474 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.651505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.653501 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.654100 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.654108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.654109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.654240 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.654529 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.654533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.654535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.654825 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.655011 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.655282 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.655714 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":161,"threads_started":1}
I20260810 14:38:08.656378 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.644263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:34051 (local address 127.0.0.1:55930)
0810 14:38:08.644789 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.644839 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.644999 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.645322 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.645328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.645797 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.646005 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.646021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.646835 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.646837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.647742 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.647751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.648235 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.648239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.648353 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.649145 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.649162 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.650732 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.652833 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.652841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.652853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.653184 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.653492 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.653495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.653497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.653690 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.654242 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.654249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.654619 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.654843 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.655261 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":129,"threads_started":1}
I20260810 14:38:08.657775 4174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:38:08.661211 4089 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } attempt: 3
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:38:08.667448 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.658355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37629 (local address 127.0.0.1:45192)
0810 14:38:08.658539 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.658554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.658638 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.658888 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.658890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.659081 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.659252 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.659259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.660002 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.660007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.660537 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.660545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.661006 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.661010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.661111 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.661891 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.661902 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.663706 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.665594 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.665597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.665599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.665866 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.666242 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.666245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.666247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.666357 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.666835 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.666840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.666952 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.667065 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.667258 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 14:38:08.667770 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.658476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.21.43.125:37629)
0810 14:38:08.658637 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.658642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.658658 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:38:08.658729 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.658732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.658776 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.658873 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.659372 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.659891 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.660659 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.660888 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.661000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.663853 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.663856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.663859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.663885 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.665483 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.665996 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.666000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.666002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.666119 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.666468 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.666472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.666473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.666718 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.666833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.667199 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.667326 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:38:08.669174 4174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:38:08.669710 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.658928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.21.43.126:34051)
0810 14:38:08.659062 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.659066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.659078 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:38:08.659119 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.659121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.659202 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.659295 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.659915 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.660405 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.661185 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.661489 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.661591 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.665642 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.665649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.665655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.665690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.667470 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.668033 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.668038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.668039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.668136 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.668470 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.668474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.668479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.668717 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.668977 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.669366 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.669504 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:38:08.671406 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.661661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53428)
0810 14:38:08.661796 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.661814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.661896 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.662299 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.662305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.662468 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.662586 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.662593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.663496 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.663499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.664199 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.664209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.665108 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.665111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.665175 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.665717 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.665730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.667193 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.669110 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.669112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.669114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.669344 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.670335 (+ 991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.670339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.670341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.670432 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.670856 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.670858 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.670941 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.671061 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.671233 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:38:08.671868 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.661895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.21.43.124:35393)
0810 14:38:08.662088 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.662092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.662105 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:38:08.662143 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.662145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.662192 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.662272 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.662748 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.663399 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.664342 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.664952 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.665065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.667334 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.667337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.667340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.667363 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.669015 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.669462 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.669468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.669470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.670232 (+ 762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.670538 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.670542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.670544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.670744 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.670850 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.671646 (+ 796us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.671766 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:38:08.674988 4089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: Bootstrap starting.
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:38:08.678164 4089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:08.679045 4089 log.cc:824] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:08.681201 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.671232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.21.43.125:37629)
0810 14:38:08.671347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.671352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.671364 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:38:08.671398 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.671401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.671445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.671660 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.672490 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.673220 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.674106 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.674791 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.674874 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.677831 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.677837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.677842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.677877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.679652 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.680105 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.680108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.680109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.680177 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.680443 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.680445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.680446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.680621 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.680758 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.680989 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.681083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:38:08.681567 4089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: No bootstrap required, opened a new log
I20260810 14:38:08.683586 4009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } attempt: 2
I20260810 14:38:08.684526 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.669700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53432)
0810 14:38:08.669926 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.669942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.670019 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.670898 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.670901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.671081 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.671213 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.671220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.672705 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.672708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.673458 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.673467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.674074 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.674081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.674164 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.674685 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.674699 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.676341 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.678639 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.678642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.678645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.678902 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.683249 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.683253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.683255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.683361 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.683976 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.683979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.684092 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.684191 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.684357 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260810 14:38:08.684912 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:38:08.685081 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.669925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.21.43.124:35393)
0810 14:38:08.670310 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.670319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.670335 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:38:08.670370 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.670372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.670421 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.670530 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.671331 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.672600 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.673583 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.673930 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.674053 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.676505 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.676509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.676512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.676544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.678510 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.679780 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.679784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.679786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.679887 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.683470 (+ 3583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.683475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.683477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.683833 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.683939 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.684765 (+ 826us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.684901 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":167,"threads_started":1}
I20260810 14:38:08.686146 4089 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.686388 4089 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:08.686453 4089 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1754f188183443caff9a2711e64b7ce, State: Initialized, Role: FOLLOWER
I20260810 14:38:08.686966 4089 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.688320 4174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: Bootstrap starting.
I20260810 14:38:08.688479 4089 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 14:38:08 dist-test-slave-chsk krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372688, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:38:08.692075 4174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:08.693352 4174 log.cc:824] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:08.695492 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.683972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:35393 (local address 127.0.0.1:53440)
0810 14:38:08.684236 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:08.684251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:08.684331 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:08.685369 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:08.685372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:08.685572 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:08.685684 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.685692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.686934 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.686937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.687860 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:08.687872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.688510 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.688514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:08.688602 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.689195 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:08.689214 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:08.690944 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:08.693646 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.693649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.693651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.693910 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.694260 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:08.694264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:08.694266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:08.694361 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:08.694830 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:08.694835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:08.694968 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 14:38:08.695073 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:38:08.695322 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 14:38:08.696051 4187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.696223 4187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.693500 4016 master_runner.cc:186] Error getting master registration for 127.21.43.126:34051: OK, instance_id { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" instance_seqno: 1786372688149971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260810 14:38:08.696439 4202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 14:38:08.696502 4016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:38:08.696823 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.684518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.21.43.124:35393)
0810 14:38:08.684877 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.684882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.684895 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:38:08.684927 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.684930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.685020 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.685801 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.685886 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.686807 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.688016 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.688318 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.691373 (+ 3055us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.691492 (+ 119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.691495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.691497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.691517 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.693517 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.694041 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.694045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.694047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.694146 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.694476 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.694479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.694480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.694702 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.696162 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.696373 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.696670 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
W20260810 14:38:08.696506 4202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:38:08.700435 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: Bootstrap starting.
I20260810 14:38:08.703642 4009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:08.704056 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.687483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.21.43.126:34051)
0810 14:38:08.687611 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.687615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.689165 (+ 1550us) server_negotiation.cc:407] Connection header received
0810 14:38:08.689344 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.689347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.689391 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.689440 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.691830 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.692596 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.697025 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.697351 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.697438 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.699354 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.699358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.699361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.699386 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.701643 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.702497 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.702501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.702503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.702597 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.702945 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.702951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.702953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.703238 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.703368 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.703738 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.703879 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:38:08.704952 4009 log.cc:824] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:08.705816 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.688129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.21.43.125:37629)
0810 14:38:08.688230 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.688234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.690592 (+ 2358us) server_negotiation.cc:407] Connection header received
0810 14:38:08.690755 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.690758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.690804 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.690852 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.696629 (+ 5777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.697354 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.698636 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.699034 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.699116 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.701007 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.701012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.701014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.701047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.702861 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.703381 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.703386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.703388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.703478 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.703801 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.703804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.703806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.704013 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.704151 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.705525 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.705653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:38:08.706616 4174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: No bootstrap required, opened a new log
I20260810 14:38:08.709019 4009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: No bootstrap required, opened a new log
I20260810 14:38:08.710942 4174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.711261 4174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:08.711316 4174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32da32697a494304a91e9f4c280d3680, State: Initialized, Role: FOLLOWER
I20260810 14:38:08.711809 4174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.713060 4009 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.713277 4009 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:08.713327 4009 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1818ef1a594348aea5e7779870912a, State: Initialized, Role: FOLLOWER
I20260810 14:38:08.713320 4174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:08.713829 4009 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.714392 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:08.688886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.21.43.124:35393)
0810 14:38:08.691112 (+ 2226us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:08.691120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:08.696853 (+ 5733us) server_negotiation.cc:407] Connection header received
0810 14:38:08.696964 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:08.696967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:08.697013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:08.697103 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:08.699248 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.699995 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.707279 (+ 7284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:08.707698 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:08.707770 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:08.709480 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:08.709485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:08.709488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:08.709515 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:08.711676 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.712193 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.712197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.712198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.712293 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:08.712620 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:08.712624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:08.712626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:08.712864 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:08.713004 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:08.713863 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 14:38:08.714004 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":139,"threads_started":1}
I20260810 14:38:08.715997 4009 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:08.720233 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.720393 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.718379 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.720518 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:38:08.721380 4231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:38:08.721457 4231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:38:08.717166 4096 master_runner.cc:186] Error getting master registration for 127.21.43.126:34051: OK, instance_id { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" instance_seqno: 1786372688149971 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:38:08.722293 4096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:38:08.726184 3932 master_runner.cc:263] No leader master found from master bf1818ef1a594348aea5e7779870912a
I20260810 14:38:08.726279 3932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 14:38:08.726599 4220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:38:08.726682 4220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:38:08.765702 4187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:08.765923 4187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.767000 4187 leader_election.cc:288] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf1818ef1a594348aea5e7779870912a (127.21.43.126:34051), 32da32697a494304a91e9f4c280d3680 (127.21.43.124:35393)
I20260810 14:38:08.767927 4142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1754f188183443caff9a2711e64b7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32da32697a494304a91e9f4c280d3680" is_pre_election: true
I20260810 14:38:08.768271 4142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1754f188183443caff9a2711e64b7ce in term 0.
I20260810 14:38:08.769121 3977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1754f188183443caff9a2711e64b7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1818ef1a594348aea5e7779870912a" is_pre_election: true
I20260810 14:38:08.769434 3977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1754f188183443caff9a2711e64b7ce in term 0.
I20260810 14:38:08.770008 4035 leader_election.cc:302] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1754f188183443caff9a2711e64b7ce, bf1818ef1a594348aea5e7779870912a; no voters:
I20260810 14:38:08.770624 4187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:08.770713 4187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:08.770754 4187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:08.772291 4187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.772682 4187 leader_election.cc:288] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [CANDIDATE]: Term 1 election: Requested vote from peers bf1818ef1a594348aea5e7779870912a (127.21.43.126:34051), 32da32697a494304a91e9f4c280d3680 (127.21.43.124:35393)
I20260810 14:38:08.773232 4142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1754f188183443caff9a2711e64b7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32da32697a494304a91e9f4c280d3680"
I20260810 14:38:08.773384 4142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:08.774767 4142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1754f188183443caff9a2711e64b7ce in term 1.
I20260810 14:38:08.775383 3977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1754f188183443caff9a2711e64b7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1818ef1a594348aea5e7779870912a"
I20260810 14:38:08.775538 3977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:08.776851 3977 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1754f188183443caff9a2711e64b7ce in term 1.
I20260810 14:38:08.777293 4035 leader_election.cc:302] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1754f188183443caff9a2711e64b7ce, bf1818ef1a594348aea5e7779870912a; no voters:
I20260810 14:38:08.777664 4187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:08.777957 4187 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [term 1 LEADER]: Becoming Leader. State: Replica: a1754f188183443caff9a2711e64b7ce, State: Running, Role: LEADER
I20260810 14:38:08.780313 4187 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } }
I20260810 14:38:08.783862 4187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1754f188183443caff9a2711e64b7ce. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.784063 4187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:08.784502 4236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:08.785686 4236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:08.793752 4142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [term 1 FOLLOWER]: Refusing update from remote peer a1754f188183443caff9a2711e64b7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:08.794670 4234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:08.802644 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1754f188183443caff9a2711e64b7ce. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.802791 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.804447 3977 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [term 1 FOLLOWER]: Refusing update from remote peer a1754f188183443caff9a2711e64b7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:08.805186 4234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:08.810202 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1754f188183443caff9a2711e64b7ce. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.810350 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.820699 4234 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.820868 4234 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:08.822381 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.822526 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.823040 4237 mvcc.cc:205] Tried to move back new op lower bound from 7316982533290270720 to 7316982533249855488. Current Snapshot: MvccSnapshot[applied={T|T < 7316982533290270720}]
I20260810 14:38:08.825083 4187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.825227 4187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:08.826438 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1754f188183443caff9a2711e64b7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1818ef1a594348aea5e7779870912a" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34051 } } peers { permanent_uuid: "a1754f188183443caff9a2711e64b7ce" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37629 } } peers { permanent_uuid: "32da32697a494304a91e9f4c280d3680" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 35393 } } }
I20260810 14:38:08.826579 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:38:08.827436 4241 mvcc.cc:205] Tried to move back new op lower bound from 7316982533290270720 to 7316982533249855488. Current Snapshot: MvccSnapshot[applied={T|T < 7316982533290270720}]
I20260810 14:38:08.831578 4238 mvcc.cc:205] Tried to move back new op lower bound from 7316982533290270720 to 7316982533249855488. Current Snapshot: MvccSnapshot[applied={T|T < 7316982533290270720}]
I20260810 14:38:08.835527 4236 catalog_manager.cc:1347] Generated new cluster ID: faa8f24de71b42f28e32e408d7143caa
I20260810 14:38:08.835605 4236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:08.865615 4236 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:38:08.867089 4236 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:08.887980 4236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1754f188183443caff9a2711e64b7ce: Generated new TSK 0
I20260810 14:38:08.889008 4236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:38:09.723423 4231 catalog_manager.cc:1259] Loaded cluster ID: faa8f24de71b42f28e32e408d7143caa
I20260810 14:38:09.723587 4231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: loading cluster ID for follower catalog manager: success
I20260810 14:38:09.725873 4231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: acquiring CA information for follower catalog manager: success
I20260810 14:38:09.727367 4231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf1818ef1a594348aea5e7779870912a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:38:09.728266 4220 catalog_manager.cc:1259] Loaded cluster ID: faa8f24de71b42f28e32e408d7143caa
I20260810 14:38:09.728358 4220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: loading cluster ID for follower catalog manager: success
I20260810 14:38:09.731210 4220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: acquiring CA information for follower catalog manager: success
I20260810 14:38:09.733093 4220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32da32697a494304a91e9f4c280d3680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:38:09.965899 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 3932
I20260810 14:38:09.978717 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4016
I20260810 14:38:09.991079 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4096
2026-08-10T14:38:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3314 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:38:10 dist-test-slave-chsk krb5kdc[4254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:38:10 dist-test-slave-chsk krb5kdc[4254](info): set up 2 sockets
Aug 10 14:38:10 dist-test-slave-chsk krb5kdc[4254](info): commencing operation
krb5kdc: starting...
W20260810 14:38:11.119244 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:38:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:38:11.263025 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39821
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:45359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:11.384347 4268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:11.384703 4268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:11.384788 4268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:11.389118 4268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:11.389237 4268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:11.389273 4268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:11.389314 4268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:11.389371 4268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:11.395345 4268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:45359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:11.396588 4268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:11.397710 4268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:38:11.406878 4268 server_base.cc:1034] running on GCE node
W20260810 14:38:11.406893 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:11.406913 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:11.406849 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:11.407876 4268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:11.409159 4268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:11.410388 4268 hybrid_clock.cc:648] HybridClock initialized: now 1786372691410353 us; error 53 us; skew 500 ppm
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372691, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:11.413532 4268 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:11.414554 4268 webserver.cc:460] Webserver started at http://127.21.43.126:45825/ using document root <none> and password file <none>
I20260810 14:38:11.415227 4268 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:11.415326 4268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:11.415654 4268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:11.418049 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "5913f9350e464ce0a57713ae9ac278d4"
format_stamp: "Formatted at 2026-08-10 14:38:11 on dist-test-slave-chsk"
server_key: "bf1ac23d17134dc8be24228bf49024d1"
server_key_iv: "a3f603bcb627b2b4cc3db40e65902682"
server_key_version: "encryptionkey@0"
I20260810 14:38:11.418844 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "5913f9350e464ce0a57713ae9ac278d4"
format_stamp: "Formatted at 2026-08-10 14:38:11 on dist-test-slave-chsk"
server_key: "bf1ac23d17134dc8be24228bf49024d1"
server_key_iv: "a3f603bcb627b2b4cc3db40e65902682"
server_key_version: "encryptionkey@0"
I20260810 14:38:11.423110 4268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:38:11.425884 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:11.427594 4268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:38:11.427770 4268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "5913f9350e464ce0a57713ae9ac278d4"
format_stamp: "Formatted at 2026-08-10 14:38:11 on dist-test-slave-chsk"
server_key: "bf1ac23d17134dc8be24228bf49024d1"
server_key_iv: "a3f603bcb627b2b4cc3db40e65902682"
server_key_version: "encryptionkey@0"
I20260810 14:38:11.427915 4268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:11.442435 4268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:11.446751 4268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:11.447032 4268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:11.455806 4268 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:39821
I20260810 14:38:11.455905 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:39821 every 8 connection(s)
I20260810 14:38:11.457008 4268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:11.460140 4340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:11.461686 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4268
I20260810 14:38:11.461803 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:38:11.462145 21677 external_mini_cluster.cc:1428] Setting key 9530e8173d3967e2940e08a1deba0efb
I20260810 14:38:11.467496 4340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4: Bootstrap starting.
I20260810 14:38:11.470520 4340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:11.471436 4340 log.cc:824] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:11.473963 4340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4: No bootstrap required, opened a new log
I20260810 14:38:11.478127 4340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } }
I20260810 14:38:11.478371 4340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:11.478406 4340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5913f9350e464ce0a57713ae9ac278d4, State: Initialized, Role: FOLLOWER
I20260810 14:38:11.478531 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:11.463724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.21.43.126:39821)
0810 14:38:11.464225 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:11.464247 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:11.464298 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:38:11.465168 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:11.465195 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:11.465646 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:11.466047 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:11.466800 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:11.468312 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:11.469253 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:11.470000 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:11.470178 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:11.473288 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:11.473294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:11.473310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:11.473340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:11.475388 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:11.475918 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:11.475924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:11.475929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:11.476060 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:11.476374 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:11.476382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:11.476383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:11.476831 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:11.477091 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:11.477331 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 14:38:11.477655 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":130,"threads_started":1}
I20260810 14:38:11.478997 4340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } }
I20260810 14:38:11.479133 4340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:11.479254 4340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:11.479390 4340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:11.480432 4340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } }
I20260810 14:38:11.480834 4340 leader_election.cc:302] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5913f9350e464ce0a57713ae9ac278d4; no voters:
I20260810 14:38:11.481249 4340 leader_election.cc:288] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:38:11.481338 4345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:11.481643 4345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5913f9350e464ce0a57713ae9ac278d4, State: Running, Role: LEADER
I20260810 14:38:11.481714 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:38:11.482002 4345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } }
I20260810 14:38:11.482703 4340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:11.483456 4347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5913f9350e464ce0a57713ae9ac278d4. Latest consensus state: current_term: 1 leader_uuid: "5913f9350e464ce0a57713ae9ac278d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } } }
I20260810 14:38:11.483608 4347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:11.483942 4346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5913f9350e464ce0a57713ae9ac278d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913f9350e464ce0a57713ae9ac278d4" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39821 } } }
I20260810 14:38:11.484109 4346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:11.484447 4352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:11.487901 4352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:11.494035 4352 catalog_manager.cc:1347] Generated new cluster ID: a2b0f15ff4b7479ab57661942af4b0d5
I20260810 14:38:11.494132 4352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:11.516979 4352 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:38:11.518256 4352 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:11.530213 4352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5913f9350e464ce0a57713ae9ac278d4: Generated new TSK 0
I20260810 14:38:11.531051 4352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:38:11.538188 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4268
2026-08-10T14:38:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1542 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4370](info): set up 2 sockets
Aug 10 14:38:11 dist-test-slave-chsk krb5kdc[4370](info): commencing operation
krb5kdc: starting...
W20260810 14:38:12.651736 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:38:12 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:38:12.798297 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:12.920704 4384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:12.921132 4384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:12.921357 4384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:12.925864 4384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:12.925979 4384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:12.926020 4384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:12.926059 4384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:12.926095 4384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:12.931908 4384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:12.933172 4384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:12.934314 4384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:12.943497 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:12.943580 4384 server_base.cc:1034] running on GCE node
W20260810 14:38:12.943594 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:12.943608 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:12.945892 4384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:12.947340 4384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:12.948570 4384 hybrid_clock.cc:648] HybridClock initialized: now 1786372692948551 us; error 46 us; skew 500 ppm
Aug 10 14:38:12 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:12.951861 4384 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:12.952873 4384 webserver.cc:460] Webserver started at http://127.21.43.126:38665/ using document root <none> and password file <none>
I20260810 14:38:12.953526 4384 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:12.953600 4384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:12.953845 4384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:12.956209 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "4ab4f40bf5594ca7a3f1ce3272180199"
format_stamp: "Formatted at 2026-08-10 14:38:12 on dist-test-slave-chsk"
server_key: "b039512821845cb217313731562ac17a"
server_key_iv: "2125ce841e129ce1afd0638be52a8d71"
server_key_version: "encryptionkey@0"
I20260810 14:38:12.956825 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "4ab4f40bf5594ca7a3f1ce3272180199"
format_stamp: "Formatted at 2026-08-10 14:38:12 on dist-test-slave-chsk"
server_key: "b039512821845cb217313731562ac17a"
server_key_iv: "2125ce841e129ce1afd0638be52a8d71"
server_key_version: "encryptionkey@0"
I20260810 14:38:12.961238 4384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 14:38:12.964211 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:12.965818 4384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:38:12.966022 4384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "4ab4f40bf5594ca7a3f1ce3272180199"
format_stamp: "Formatted at 2026-08-10 14:38:12 on dist-test-slave-chsk"
server_key: "b039512821845cb217313731562ac17a"
server_key_iv: "2125ce841e129ce1afd0638be52a8d71"
server_key_version: "encryptionkey@0"
I20260810 14:38:12.966158 4384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:12.984254 4384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:12.987869 4384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:12.988118 4384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:12.996196 4384 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40161
I20260810 14:38:12.996529 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40161 every 8 connection(s)
I20260810 14:38:12.997434 4384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:13.000546 4454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:13.006704 4454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap starting.
I20260810 14:38:13.007627 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4384
I20260810 14:38:13.007812 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:38:13.008267 21677 external_mini_cluster.cc:1428] Setting key 9a137b020bae76983d1b1d1b7c00eb50
I20260810 14:38:13.010075 4454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:13.010951 4454 log.cc:824] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:13.013062 4454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: No bootstrap required, opened a new log
I20260810 14:38:13.017233 4454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:13.017491 4454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:13.017561 4454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Initialized, Role: FOLLOWER
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:13.018075 4454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:13.018278 4454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:13.018364 4454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:13.018507 4454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:13.019912 4454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:13.020313 4454 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab4f40bf5594ca7a3f1ce3272180199; no voters:
I20260810 14:38:13.020700 4454 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:38:13.020782 4462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:13.020989 4462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 LEADER]: Becoming Leader. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Running, Role: LEADER
I20260810 14:38:13.021382 4462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:13.022184 4454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:13.022706 4462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab4f40bf5594ca7a3f1ce3272180199. Latest consensus state: current_term: 1 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:13.022894 4462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:13.023515 4463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:13.023653 4463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:13.024507 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.009707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.21.43.126:40161)
0810 14:38:13.010267 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.010284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.010321 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:38:13.011003 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.011046 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.011479 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.011833 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:13.012695 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.013753 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.014656 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.015247 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.015414 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.018352 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:13.018357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:13.018371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:13.018428 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:13.020500 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.021141 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.021148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.021154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.021299 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.021643 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.021647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.021649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.022149 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:13.022382 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.022689 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.023040 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":159,"threads_started":1}
I20260810 14:38:13.024641 4468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:13.028739 4468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:13.035544 4468 catalog_manager.cc:1347] Generated new cluster ID: f18227f4d2e4401aada656440c05463f
I20260810 14:38:13.035647 4468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:13.056385 4468 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:38:13.057787 4468 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:13.074819 4468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Generated new TSK 0
I20260810 14:38:13.075829 4468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:38:13.111465 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:13.234807 4484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:13.235200 4484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:13.235287 4484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:13.239574 4484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:13.239689 4484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:13.239790 4484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:13.245450 4484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:13.246708 4484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:13.247850 4484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:13.258312 4491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:13.258344 4490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:13.258328 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:13.258509 4484 server_base.cc:1034] running on GCE node
I20260810 14:38:13.259124 4484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:13.260082 4484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:13.261279 4484 hybrid_clock.cc:648] HybridClock initialized: now 1786372693261258 us; error 36 us; skew 500 ppm
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:13.264638 4484 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:13.265643 4484 webserver.cc:460] Webserver started at http://127.21.43.65:37697/ using document root <none> and password file <none>
I20260810 14:38:13.266208 4484 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:13.266270 4484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:13.266448 4484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:13.268610 4484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "0d43ab2307cc4d91b14971a1ed414fea"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "da408b11dac915fac0bd53fe827052cc"
server_key_iv: "88094459bc33c722665298d5bfb1d162"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.269166 4484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "0d43ab2307cc4d91b14971a1ed414fea"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "da408b11dac915fac0bd53fe827052cc"
server_key_iv: "88094459bc33c722665298d5bfb1d162"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.273491 4484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:38:13.276038 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.277410 4484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:13.277556 4484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "0d43ab2307cc4d91b14971a1ed414fea"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "da408b11dac915fac0bd53fe827052cc"
server_key_iv: "88094459bc33c722665298d5bfb1d162"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.277681 4484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:13.288242 4484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:13.291389 4484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:13.291604 4484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:13.292248 4484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:13.293136 4484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:13.293185 4484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.293256 4484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:13.293289 4484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.302881 4484 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:42273
I20260810 14:38:13.302899 4626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:42273 every 8 connection(s)
I20260810 14:38:13.303974 4484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:13.310472 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4484
I20260810 14:38:13.310668 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:38:13.311003 21677 external_mini_cluster.cc:1428] Setting key f06aa13bf0e33fd0ea9779d4a85a78e6
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:13.320238 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.306029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:51893 (local address 127.21.43.126:40161)
0810 14:38:13.306178 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.306182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.307087 (+ 905us) server_negotiation.cc:407] Connection header received
0810 14:38:13.308778 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.308782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.308837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.308923 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:13.310214 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.310893 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.312308 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.312654 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.312744 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.315898 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:13.315902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:13.315904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:13.315930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:13.317688 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.318462 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.318471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.318472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.318569 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.318942 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.318945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.318947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.319224 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:13.319335 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.319893 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.320055 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:38:13.321136 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.306423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:51893)
0810 14:38:13.306918 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.306964 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.308483 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.309105 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.309112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.309510 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:13.310035 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.310051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.311052 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.311056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.312174 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.312185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.312837 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.312842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.313078 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.313807 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:13.313834 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:13.315697 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:13.317986 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.317993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.318018 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.318331 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.318671 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.318676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.318678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.318834 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.319347 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:13.319355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:13.319638 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.319834 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.320197 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:38:13.322679 4627 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:13.322956 4627 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:13.323639 4627 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:13.325420 4416 ts_manager.cc:194] Registered new tserver with Master: 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:38:13.326792 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:51893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:38:13.343055 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:13.462378 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:13.462677 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:13.462724 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:13.467038 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:13.467118 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:13.467254 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:13.473232 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:13.474408 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:13.475576 4632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:13.485989 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:13.486061 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:13.486116 4632 server_base.cc:1034] running on GCE node
W20260810 14:38:13.486042 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:13.486680 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:13.487454 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:13.488691 4632 hybrid_clock.cc:648] HybridClock initialized: now 1786372693488675 us; error 29 us; skew 500 ppm
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:13.491954 4632 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:13.493029 4632 webserver.cc:460] Webserver started at http://127.21.43.66:35707/ using document root <none> and password file <none>
I20260810 14:38:13.493661 4632 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:13.493759 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:13.494016 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:13.496301 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "70167493f55ca066de12b63066ee1578"
server_key_iv: "d88f88b442ff0915395e68c61a07916b"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.496927 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "70167493f55ca066de12b63066ee1578"
server_key_iv: "d88f88b442ff0915395e68c61a07916b"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.501071 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 14:38:13.503888 4647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.505247 4632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:38:13.505395 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "70167493f55ca066de12b63066ee1578"
server_key_iv: "d88f88b442ff0915395e68c61a07916b"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.505517 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:13.525234 4632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:13.528683 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:13.528909 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:13.529575 4632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:13.530479 4632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:13.530551 4632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.530622 4632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:13.530668 4632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.540011 4632 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:40739
I20260810 14:38:13.540035 4772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:40739 every 8 connection(s)
I20260810 14:38:13.541060 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:13.541401 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4632
I20260810 14:38:13.541531 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:38:13.541849 21677 external_mini_cluster.cc:1428] Setting key 5a3c5eb9df768a4cf4389c1a4cc43f52
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:38:13.560312 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.543087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:53379 (local address 127.21.43.126:40161)
0810 14:38:13.543259 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.543264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.544417 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 14:38:13.546452 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.546455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.546503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.546597 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:13.549144 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.550102 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.551785 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.552225 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.552369 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.555121 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:13.555124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:13.555126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:13.555201 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:13.557441 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.558200 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.558204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.558205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.558298 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.558689 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.558693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.558695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.558913 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:13.559023 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.560038 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.560161 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:38:13.561513 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.543601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:53379)
0810 14:38:13.544147 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.544212 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.546148 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.546837 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.546849 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.547473 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:13.548685 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.548713 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.550417 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.550422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.551610 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.551626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.552444 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.552447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.552633 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.553338 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:13.553357 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:13.554902 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:13.557720 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.557726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.557743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.558056 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.558401 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.558404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.558406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.558574 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.559039 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:13.559047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:13.559652 (+ 605us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.559923 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.560362 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":142,"threads_started":1}
I20260810 14:38:13.562769 4773 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:13.563076 4773 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:13.563779 4773 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:13.565059 4416 ts_manager.cc:194] Registered new tserver with Master: a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739)
I20260810 14:38:13.565685 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:53379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:38:13.578424 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:13.698189 4778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:13.698535 4778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:13.698619 4778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:13.702858 4778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:13.702979 4778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:13.703086 4778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:13.708825 4778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:13.710232 4778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:13.711383 4778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:13.722431 4785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:13.722375 4784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:13.722375 4787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:13.722543 4778 server_base.cc:1034] running on GCE node
I20260810 14:38:13.723433 4778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:13.724341 4778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:13.725636 4778 hybrid_clock.cc:648] HybridClock initialized: now 1786372693725594 us; error 36 us; skew 500 ppm
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:13.731428 4778 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:13.733664 4778 webserver.cc:460] Webserver started at http://127.21.43.67:45403/ using document root <none> and password file <none>
I20260810 14:38:13.734629 4778 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:13.734757 4778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:13.735064 4778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:13.737762 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "d7bd93813fe745949c7ae930715c0105"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "068afb18640841ba6460f0e9983aae63"
server_key_iv: "f32aafaaf87e57b10679155df12e6e1d"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.738350 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "d7bd93813fe745949c7ae930715c0105"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "068afb18640841ba6460f0e9983aae63"
server_key_iv: "f32aafaaf87e57b10679155df12e6e1d"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.743005 4778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:38:13.746008 4796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.747617 4778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:13.747747 4778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "d7bd93813fe745949c7ae930715c0105"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "068afb18640841ba6460f0e9983aae63"
server_key_iv: "f32aafaaf87e57b10679155df12e6e1d"
server_key_version: "encryptionkey@0"
I20260810 14:38:13.747856 4778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:13.773363 4778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:13.776969 4778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:13.777216 4778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:13.778007 4778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:13.778962 4778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:13.779038 4778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:13.779115 4778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:13.779201 4778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 14:38:13.790875 4778 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:38011
I20260810 14:38:13.790915 4920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:38011 every 8 connection(s)
I20260810 14:38:13.792449 4778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:13.797098 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4778
I20260810 14:38:13.797269 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:38:13.797587 21677 external_mini_cluster.cc:1428] Setting key 2ca0d1324e226b904e4adac3b2108449
Aug 10 14:38:13 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372693, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:13.809073 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.794773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:60125 (local address 127.21.43.126:40161)
0810 14:38:13.794944 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.794949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.796045 (+ 1096us) server_negotiation.cc:407] Connection header received
0810 14:38:13.797605 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.797610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.797685 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.797792 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:13.799478 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.800001 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.801245 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.801475 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.801590 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.804789 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:13.804792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:13.804794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:13.804823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:13.806579 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.807261 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.807265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.807266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.807350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.807729 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.807733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.807734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.807944 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:13.808056 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.808747 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.808897 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:38:13.810170 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.795325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:60125)
0810 14:38:13.795868 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.795916 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.797332 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.797955 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.797963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.798482 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:13.799131 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.799194 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.800155 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.800159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.801098 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.801110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.801652 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.801656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.801840 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.802523 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:13.802541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:13.804549 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:13.806751 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.806758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.806785 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.807096 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.807453 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:13.807456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:13.807458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:13.807616 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:13.808056 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:13.808064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:13.808445 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.808689 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.809064 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":128,"threads_started":1}
I20260810 14:38:13.811666 4921 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:13.811945 4921 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:13.812561 4921 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:13.813773 4415 ts_manager.cc:194] Registered new tserver with Master: d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
I20260810 14:38:13.814445 4415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:60125
I20260810 14:38:13.824916 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:13.836716 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.827241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.21.43.126:40161)
0810 14:38:13.827596 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.827603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.827831 (+ 228us) server_negotiation.cc:407] Connection header received
0810 14:38:13.828060 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.828064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.828129 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.828234 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:13.829170 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.829771 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.830749 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.831279 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.831470 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.833186 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:13.833189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:13.833191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:13.833224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:13.834757 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.835301 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.835305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.835309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.835394 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:13.835698 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:13.835702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:13.835703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:13.835974 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:13.836144 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.836399 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.836531 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260810 14:38:13.840431 4415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:38:13.843864 4415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:38:13.861620 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.855864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.21.43.65:42273)
0810 14:38:13.856455 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.856460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.856478 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:38:13.856546 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.856550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.856688 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.856853 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:13.857228 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.857966 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.859783 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.860718 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.860840 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.861010 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.861379 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.861481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":134,"threads_started":1}
I20260810 14:38:13.861621 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.855361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.0.0.1:50954)
0810 14:38:13.856109 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.856121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.856208 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.856998 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.857001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.857020 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:13.857109 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.857113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.858103 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.858108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.859606 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.859619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.860885 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.860889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.861091 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.861143 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.861303 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.861453 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":88,"thread_start_us":141,"threads_started":1}
I20260810 14:38:13.862663 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.854545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38011 (local address 127.0.0.1:53638)
0810 14:38:13.855015 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.855066 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.855249 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.856285 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.856292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.856319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:13.856501 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.856508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.858340 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.858343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.859733 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.859746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.862134 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.862137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.862255 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.862279 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.862356 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.862436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":25,"thread_start_us":124,"threads_started":1}
I20260810 14:38:13.863034 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.855435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.21.43.66:40739)
0810 14:38:13.856319 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.856331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.856354 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:38:13.856427 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.856431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.856618 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.856770 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:13.857038 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.858196 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.859877 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.860713 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.860840 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.861010 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.862752 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.862878 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":129,"threads_started":1}
I20260810 14:38:13.863212 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.855287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.21.43.67:38011)
0810 14:38:13.855703 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.855715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.855741 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:38:13.855824 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.855831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.856067 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.856747 (+ 680us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:13.856917 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.858196 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.861078 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.861967 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.862119 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.862295 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.862886 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.862994 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":185,"threads_started":1}
I20260810 14:38:13.862666 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.854608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:40739 (local address 127.0.0.1:57482)
0810 14:38:13.855815 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.855830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.855917 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.856784 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.856789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.856811 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:13.856910 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.856916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.858481 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.858484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.859739 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.859746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.860885 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.860889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.862407 (+ 1518us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.862427 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.862492 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.862552 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":1380,"thread_start_us":143,"threads_started":1}
I20260810 14:38:13.865356 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 221cbb26a3ad44f587b0be87932898ea (DEFAULT_TABLE table=test-table [id=2eaad48f0d644cb7be793fdf1153b253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:13.866546 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cbb26a3ad44f587b0be87932898ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:13.865986 4700 tablet_service.cc:1467] Processing CreateTablet for tablet 221cbb26a3ad44f587b0be87932898ea (DEFAULT_TABLE table=test-table [id=2eaad48f0d644cb7be793fdf1153b253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:13.867281 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cbb26a3ad44f587b0be87932898ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:13.867183 4850 tablet_service.cc:1467] Processing CreateTablet for tablet 221cbb26a3ad44f587b0be87932898ea (DEFAULT_TABLE table=test-table [id=2eaad48f0d644cb7be793fdf1153b253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:13.868224 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 221cbb26a3ad44f587b0be87932898ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:13.875497 4941 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap starting.
I20260810 14:38:13.876493 4945 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Bootstrap starting.
I20260810 14:38:13.879307 4941 tablet_bootstrap.cc:654] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:13.879307 4945 tablet_bootstrap.cc:654] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:13.880429 4941 log.cc:824] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:13.880649 4945 log.cc:824] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:13.881959 4946 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Bootstrap starting.
I20260810 14:38:13.882928 4941 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: No bootstrap required, opened a new log
I20260810 14:38:13.882992 4945 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: No bootstrap required, opened a new log
I20260810 14:38:13.883255 4941 ts_tablet_manager.cc:1397] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260810 14:38:13.883245 4945 ts_tablet_manager.cc:1397] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260810 14:38:13.885443 4946 tablet_bootstrap.cc:654] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:13.886626 4946 log.cc:824] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:13.887712 4941 raft_consensus.cc:348] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.887686 4945 raft_consensus.cc:348] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.888032 4941 raft_consensus.cc:374] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:13.888039 4945 raft_consensus.cc:374] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:13.888094 4941 raft_consensus.cc:729] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d43ab2307cc4d91b14971a1ed414fea, State: Initialized, Role: FOLLOWER
I20260810 14:38:13.888125 4945 raft_consensus.cc:729] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bd93813fe745949c7ae930715c0105, State: Initialized, Role: FOLLOWER
I20260810 14:38:13.888612 4941 consensus_queue.cc:260] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.888624 4945 consensus_queue.cc:260] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.889407 4921 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
I20260810 14:38:13.889516 4946 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: No bootstrap required, opened a new log
I20260810 14:38:13.889658 4627 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
I20260810 14:38:13.889753 4946 ts_tablet_manager.cc:1397] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 14:38:13.889818 4945 ts_tablet_manager.cc:1428] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:38:13.890350 4941 ts_tablet_manager.cc:1428] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260810 14:38:13.894065 4946 raft_consensus.cc:348] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.894438 4946 raft_consensus.cc:374] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:13.894510 4946 raft_consensus.cc:729] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4408bbf3c30414c8a56d849a83f3b7d, State: Initialized, Role: FOLLOWER
I20260810 14:38:13.895057 4946 consensus_queue.cc:260] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.895911 4773 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
I20260810 14:38:13.896261 4946 ts_tablet_manager.cc:1428] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 14:38:13.987737 4953 raft_consensus.cc:482] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:13.987972 4953 raft_consensus.cc:504] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.989204 4953 leader_election.cc:288] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011), 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
I20260810 14:38:13.993700 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.989751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:42457 (local address 127.21.43.67:38011)
0810 14:38:13.989914 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.989920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.989942 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:38:13.990062 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.990065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.990120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.990218 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:13.990640 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.991445 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.992636 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.993088 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.993247 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.993308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.993488 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.993560 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:38:13.993700 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.989643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38011 (local address 127.21.43.66:42457)
0810 14:38:13.989797 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.989814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.989926 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.990328 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.990331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.990352 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:13.990505 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.990514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.991592 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.991595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.992499 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.992508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.993240 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.993244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.993360 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.993379 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.993469 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.993547 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":26}
I20260810 14:38:13.993834 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.990138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:44227 (local address 127.21.43.65:42273)
0810 14:38:13.990273 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:13.990277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:13.990441 (+ 164us) server_negotiation.cc:407] Connection header received
0810 14:38:13.990542 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:13.990544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:13.990616 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:13.990706 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:13.991008 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.991706 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.992795 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.993219 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.993353 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.993408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:13.993604 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 14:38:13.993678 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:38:13.993891 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:13.989967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.21.43.66:44227)
0810 14:38:13.990343 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:13.990355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:13.990449 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:13.990765 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:13.990767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:13.990785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:13.990886 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.990890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.991828 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.991832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.992658 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:13.992668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:13.993351 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:13.993355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:13.993463 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:13.993485 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:38:13.993585 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:38:13.993661 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":21,"thread_start_us":166,"threads_started":1}
I20260810 14:38:13.994683 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cbb26a3ad44f587b0be87932898ea" candidate_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d43ab2307cc4d91b14971a1ed414fea" is_pre_election: true
I20260810 14:38:13.994673 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cbb26a3ad44f587b0be87932898ea" candidate_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bd93813fe745949c7ae930715c0105" is_pre_election: true
I20260810 14:38:13.995028 4575 raft_consensus.cc:2454] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4408bbf3c30414c8a56d849a83f3b7d in term 0.
I20260810 14:38:13.995029 4872 raft_consensus.cc:2454] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4408bbf3c30414c8a56d849a83f3b7d in term 0.
I20260810 14:38:13.995697 4651 leader_election.cc:302] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea, a4408bbf3c30414c8a56d849a83f3b7d; no voters:
I20260810 14:38:13.996057 4953 raft_consensus.cc:2781] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:13.996169 4953 raft_consensus.cc:482] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:13.996232 4953 raft_consensus.cc:3037] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:13.997359 4953 raft_consensus.cc:504] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:13.997759 4953 leader_election.cc:288] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [CANDIDATE]: Term 1 election: Requested vote from peers d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011), 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
I20260810 14:38:13.998255 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cbb26a3ad44f587b0be87932898ea" candidate_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d43ab2307cc4d91b14971a1ed414fea"
I20260810 14:38:13.998267 4872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "221cbb26a3ad44f587b0be87932898ea" candidate_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bd93813fe745949c7ae930715c0105"
I20260810 14:38:13.998422 4575 raft_consensus.cc:3037] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:13.998420 4872 raft_consensus.cc:3037] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:13.999660 4575 raft_consensus.cc:2454] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4408bbf3c30414c8a56d849a83f3b7d in term 1.
I20260810 14:38:13.999867 4872 raft_consensus.cc:2454] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4408bbf3c30414c8a56d849a83f3b7d in term 1.
I20260810 14:38:14.000020 4651 leader_election.cc:302] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea, a4408bbf3c30414c8a56d849a83f3b7d; no voters:
I20260810 14:38:14.000236 4953 raft_consensus.cc:2781] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:14.000527 4953 raft_consensus.cc:686] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 1 LEADER]: Becoming Leader. State: Replica: a4408bbf3c30414c8a56d849a83f3b7d, State: Running, Role: LEADER
I20260810 14:38:14.000916 4953 consensus_queue.cc:237] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:14.004945 4415 catalog_manager.cc:5495] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d reported cstate change: term changed from 0 to 1, leader changed from <none> to a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66). New cstate: current_term: 1 leader_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } health_report { overall_health: UNKNOWN } } }
I20260810 14:38:14.018940 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.015018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.21.43.66:40739)
0810 14:38:14.015264 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.015269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.015283 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:38:14.015484 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.015487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.015547 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.015638 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:14.015946 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.016456 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.017355 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.017576 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.017669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.017982 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:14.018496 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:14.018604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:14.018670 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 14:38:14.018726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:38:14.025499 4575 raft_consensus.cc:1261] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Refusing update from remote peer a4408bbf3c30414c8a56d849a83f3b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:14.025493 4872 raft_consensus.cc:1261] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Refusing update from remote peer a4408bbf3c30414c8a56d849a83f3b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:14.026345 4953 consensus_queue.cc:1035] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:14.026620 4955 consensus_queue.cc:1035] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:14.038098 4960 mvcc.cc:205] Tried to move back new op lower bound from 7316982554719698944 to 7316982554631229440. Current Snapshot: MvccSnapshot[applied={T|T < 7316982554719698944}]
I20260810 14:38:14.045643 4415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:14.045840 4415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:14.048528 4415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Sending DeleteTablet for 3 replicas of tablet 221cbb26a3ad44f587b0be87932898ea
I20260810 14:38:14.049504 4700 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cbb26a3ad44f587b0be87932898ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57482
I20260810 14:38:14.049758 4850 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cbb26a3ad44f587b0be87932898ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:53638
I20260810 14:38:14.049753 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cbb26a3ad44f587b0be87932898ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:50954
I20260810 14:38:14.050336 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4484
I20260810 14:38:14.053893 4970 tablet_replica.cc:330] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: stopping tablet replica
I20260810 14:38:14.054344 4970 raft_consensus.cc:2229] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:14.054776 4970 raft_consensus.cc:2258] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:14.055200 4968 tablet_replica.cc:330] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: stopping tablet replica
I20260810 14:38:14.055538 4968 raft_consensus.cc:2229] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:38:14.055943 4968 raft_consensus.cc:2258] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:14.057006 4970 ts_tablet_manager.cc:1905] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:14.057394 4968 ts_tablet_manager.cc:1905] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:14.060294 4968 ts_tablet_manager.cc:1918] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:14.060418 4968 log.cc:1196] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/221cbb26a3ad44f587b0be87932898ea
I20260810 14:38:14.060767 4968 ts_tablet_manager.cc:1939] T 221cbb26a3ad44f587b0be87932898ea P a4408bbf3c30414c8a56d849a83f3b7d: Deleting consensus metadata
I20260810 14:38:14.061151 4970 ts_tablet_manager.cc:1918] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:14.061352 4970 log.cc:1196] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/221cbb26a3ad44f587b0be87932898ea
I20260810 14:38:14.061777 4970 ts_tablet_manager.cc:1939] T 221cbb26a3ad44f587b0be87932898ea P d7bd93813fe745949c7ae930715c0105: Deleting consensus metadata
I20260810 14:38:14.062286 4400 catalog_manager.cc:4841] TS a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): tablet 221cbb26a3ad44f587b0be87932898ea (table test-table [id=2eaad48f0d644cb7be793fdf1153b253]) successfully deleted
I20260810 14:38:14.063413 4403 catalog_manager.cc:4841] TS d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): tablet 221cbb26a3ad44f587b0be87932898ea (table test-table [id=2eaad48f0d644cb7be793fdf1153b253]) successfully deleted
W20260810 14:38:14.063814 4403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:42273 (error 108)
I20260810 14:38:14.064682 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4632
I20260810 14:38:14.064965 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.064638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.0.0.1:50962)
0810 14:38:14.064783 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:14.064863 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 14:38:14.069873 4403 catalog_manager.cc:4584] TS 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 221cbb26a3ad44f587b0be87932898ea: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42273: connect: Connection refused (error 111)
I20260810 14:38:14.073973 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4778
I20260810 14:38:14.083276 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4384
I20260810 14:38:14.093334 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--webserver_interface=127.21.43.126
--webserver_port=38665
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:14.222158 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:14.222468 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:14.222513 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:14.227187 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:14.227272 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:14.227293 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:14.227310 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:14.227326 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:14.234022 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=38665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:14.235370 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:14.236567 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:14.254350 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.254354 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:14.254599 4973 server_base.cc:1034] running on GCE node
W20260810 14:38:14.254354 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:14.255805 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:14.257160 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:14.258363 4973 hybrid_clock.cc:648] HybridClock initialized: now 1786372694258330 us; error 52 us; skew 500 ppm
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:14.261569 4973 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:14.262697 4973 webserver.cc:460] Webserver started at http://127.21.43.126:38665/ using document root <none> and password file <none>
I20260810 14:38:14.263396 4973 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:14.263484 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:14.267715 4973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:38:14.270057 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.271554 4973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:14.271725 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "4ab4f40bf5594ca7a3f1ce3272180199"
format_stamp: "Formatted at 2026-08-10 14:38:12 on dist-test-slave-chsk"
server_key: "b039512821845cb217313731562ac17a"
server_key_iv: "2125ce841e129ce1afd0638be52a8d71"
server_key_version: "encryptionkey@0"
I20260810 14:38:14.272392 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:14.287711 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:14.291489 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:14.291744 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:14.299901 4973 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40161
I20260810 14:38:14.300242 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40161 every 8 connection(s)
I20260810 14:38:14.301417 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:14.302568 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 4973
I20260810 14:38:14.303062 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:42273
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=37697
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:14.307967 5048 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:38:14.311131 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap starting.
I20260810 14:38:14.335824 5048 log.cc:824] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:14.347316 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:14.347826 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap complete.
I20260810 14:38:14.353878 5048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:14.354429 5048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Initialized, Role: FOLLOWER
I20260810 14:38:14.355013 5048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:14.355190 5048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:14.355273 5048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:14.355372 5048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:38:14.358096 5048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:14.358512 5048 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab4f40bf5594ca7a3f1ce3272180199; no voters:
I20260810 14:38:14.358896 5048 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:38:14.359071 5054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:38:14.359402 5054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 LEADER]: Becoming Leader. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Running, Role: LEADER
I20260810 14:38:14.359750 5054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:14.360287 5048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:14.360805 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab4f40bf5594ca7a3f1ce3272180199. Latest consensus state: current_term: 2 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:14.360939 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:14.361428 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:14.361539 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:14.361831 5059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:14.367015 5059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:14.367939 5059 catalog_manager.cc:1259] Loaded cluster ID: f18227f4d2e4401aada656440c05463f
I20260810 14:38:14.368021 5059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:14.369786 5059 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 14:38:14.374236 5059 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 14:38:14.380260 5059 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:14.381247 5059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Loaded TSK: 0
I20260810 14:38:14.382612 5059 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:38:14.451037 5051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:14.451380 5051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:14.451468 5051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:14.455807 5051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:14.455885 5051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:14.455950 5051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:14.461927 5051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:42273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=37697
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:14.463937 5051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:14.465586 5051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:14.476339 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.476392 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.476794 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:14.477092 5051 server_base.cc:1034] running on GCE node
I20260810 14:38:14.477573 5051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:14.478539 5051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:14.480085 5051 hybrid_clock.cc:648] HybridClock initialized: now 1786372694480015 us; error 104 us; skew 500 ppm
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:14.484539 5051 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:14.485733 5051 webserver.cc:460] Webserver started at http://127.21.43.65:37697/ using document root <none> and password file <none>
I20260810 14:38:14.486382 5051 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:14.486523 5051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:14.491053 5051 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:14.493618 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.495119 5051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:38:14.495326 5051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "0d43ab2307cc4d91b14971a1ed414fea"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "da408b11dac915fac0bd53fe827052cc"
server_key_iv: "88094459bc33c722665298d5bfb1d162"
server_key_version: "encryptionkey@0"
I20260810 14:38:14.495939 5051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:14.520452 5051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:14.524029 5051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:14.524276 5051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:14.524952 5051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:14.526259 5096 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:38:14.530954 5051 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:38:14.531109 5051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 14:38:14.531276 5051 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:38:14.533154 5051 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:38:14.533239 5051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 14:38:14.533550 5096 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap starting.
I20260810 14:38:14.544909 5051 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:42273
I20260810 14:38:14.544993 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:42273 every 8 connection(s)
I20260810 14:38:14.546006 5051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:14.553350 5096 log.cc:824] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:14.553632 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5051
I20260810 14:38:14.554119 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:40739
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=35707
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:14.562793 5096 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:14.563413 5096 tablet_bootstrap.cc:492] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap complete.
I20260810 14:38:14.564795 5096 ts_tablet_manager.cc:1397] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.010s
I20260810 14:38:14.566375 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.548314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:41407)
0810 14:38:14.548917 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:14.548961 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:14.549804 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:14.550908 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:14.550916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:14.551408 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:14.551868 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.551883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.553249 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.553252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.554552 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.554564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.555749 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.555753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.555970 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.556708 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:14.556731 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:14.558966 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:14.562019 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.562027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.562044 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.562601 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.563098 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.563104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.563110 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.563348 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.564364 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:14.564374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:14.564705 (+ 331us) client_negotiation.cc:769] Sending connection context
0810 14:38:14.564968 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 14:38:14.565365 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":210,"threads_started":1}
I20260810 14:38:14.567353 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.548257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:41407 (local address 127.21.43.126:40161)
0810 14:38:14.548630 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.548651 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.549088 (+ 437us) server_negotiation.cc:407] Connection header received
0810 14:38:14.550219 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.550229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.550518 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.550854 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:14.552062 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.553068 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.554715 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.555565 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.555791 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.559276 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:14.559283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:14.559300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:14.559339 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:14.561822 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.562758 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.562764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.562769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.562937 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.563477 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.563481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.563486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.564204 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:14.564372 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:14.565144 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 14:38:14.565519 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":130,"threads_started":1}
I20260810 14:38:14.567909 5096 raft_consensus.cc:348] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:14.568445 5096 raft_consensus.cc:729] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d43ab2307cc4d91b14971a1ed414fea, State: Initialized, Role: FOLLOWER
I20260810 14:38:14.568987 5096 consensus_queue.cc:260] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:14.569860 5216 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:14.570009 5096 ts_tablet_manager.cc:1428] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 14:38:14.570122 5216 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:14.570736 5216 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:14.573822 5006 ts_manager.cc:194] Registered new tserver with Master: 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
I20260810 14:38:14.578343 5006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:41407
I20260810 14:38:14.581192 5216 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:38:14.589449 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.577636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.0.0.1:50978)
0810 14:38:14.578502 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:14.578557 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:14.578756 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:14.579417 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:14.579423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:14.579895 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:14.580035 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.580044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.581517 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.581520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.582326 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.582336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.582944 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.582947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.583059 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.583745 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:14.583758 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:14.585508 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:14.587515 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.587525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.587531 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.587787 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.588111 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.588115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.588117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.588209 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.588742 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:14.588748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:14.588882 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 14:38:14.589027 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 14:38:14.589248 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":138,"threads_started":1}
I20260810 14:38:14.589449 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.577714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.21.43.65:42273)
0810 14:38:14.578158 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.578170 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.578669 (+ 499us) server_negotiation.cc:407] Connection header received
0810 14:38:14.578919 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.578923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.579226 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.579385 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:14.580181 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.581368 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.582463 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.582818 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.582939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.585695 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:14.585705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:14.585715 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:14.585740 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:14.587389 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.587921 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.587927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.587928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.588007 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.588303 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.588308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.588310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.588617 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:14.588767 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:14.589112 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 14:38:14.589295 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":107,"threads_started":1}
I20260810 14:38:14.590628 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 221cbb26a3ad44f587b0be87932898ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:14 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:50978
I20260810 14:38:14.591703 5225 tablet_replica.cc:330] stopping tablet replica
I20260810 14:38:14.592011 5225 raft_consensus.cc:2229] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:14.592290 5225 raft_consensus.cc:2258] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:14.593338 5225 ts_tablet_manager.cc:1905] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:14.597043 5225 ts_tablet_manager.cc:1918] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:14.597184 5225 log.cc:1196] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/221cbb26a3ad44f587b0be87932898ea
I20260810 14:38:14.597458 5225 ts_tablet_manager.cc:1939] T 221cbb26a3ad44f587b0be87932898ea P 0d43ab2307cc4d91b14971a1ed414fea: Deleting consensus metadata
I20260810 14:38:14.599344 4993 catalog_manager.cc:4841] TS 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273): tablet 221cbb26a3ad44f587b0be87932898ea (table test-table [id=2eaad48f0d644cb7be793fdf1153b253]) successfully deleted
W20260810 14:38:14.685599 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:14.685894 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:14.685974 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:14.690207 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:14.690289 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:14.690384 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:14.696098 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:40739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=35707
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:14.697321 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:14.698678 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:14.709374 5231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.709374 5232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.709654 5234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:14.709654 5221 server_base.cc:1034] running on GCE node
I20260810 14:38:14.710201 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:14.710944 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:14.712138 5221 hybrid_clock.cc:648] HybridClock initialized: now 1786372694712144 us; error 62 us; skew 500 ppm
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:14.714972 5221 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:14.716002 5221 webserver.cc:460] Webserver started at http://127.21.43.66:35707/ using document root <none> and password file <none>
I20260810 14:38:14.716594 5221 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:14.716711 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:14.720762 5221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 14:38:14.723037 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.724385 5221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:14.724534 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "70167493f55ca066de12b63066ee1578"
server_key_iv: "d88f88b442ff0915395e68c61a07916b"
server_key_version: "encryptionkey@0"
I20260810 14:38:14.725164 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:14.740744 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:14.744014 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:14.744246 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:14.744899 5221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:14.745779 5221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:14.745851 5221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.745918 5221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:14.745962 5221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.755688 5221 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:40739
I20260810 14:38:14.755713 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:40739 every 8 connection(s)
I20260810 14:38:14.756772 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:14.763974 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5221
I20260810 14:38:14.764441 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:38011
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=45403
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:14.771351 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.758725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:38051 (local address 127.21.43.126:40161)
0810 14:38:14.758880 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.758883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.759758 (+ 875us) server_negotiation.cc:407] Connection header received
0810 14:38:14.761249 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.761252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.761316 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.761426 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:14.762668 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.763322 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.764509 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.764842 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.764962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.767651 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:14.767654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:14.767656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:14.767682 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:14.769121 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.769742 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.769746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.769747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.769834 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.770196 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.770200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.770202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.770389 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:14.770501 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:14.771008 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 14:38:14.771175 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:38:14.772221 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.759113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:38051)
0810 14:38:14.759610 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:14.759653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:14.761014 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:14.761640 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:14.761649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:14.762035 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:14.762520 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.762533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.763448 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.763452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.764385 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.764394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.765008 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.765012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.765223 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.765858 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:14.765875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:14.767461 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:14.769244 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.769251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.769268 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.769636 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.769929 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.769933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.769935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.770098 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.770499 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:14.770506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:14.770770 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 14:38:14.770967 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 14:38:14.771345 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260810 14:38:14.773484 5369 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:14.773753 5369 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:14.774260 5369 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:14.775445 5006 ts_manager.cc:194] Registered new tserver with Master: a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739)
I20260810 14:38:14.776374 5006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:38051
W20260810 14:38:14.886025 5372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:14.886350 5372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:14.886399 5372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:14.890589 5372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:14.890669 5372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:14.890729 5372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:14.896427 5372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:38011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=45403
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:14.897552 5372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:14.898578 5372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:14.908996 5381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.909032 5379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:14.909037 5377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:14.909191 5372 server_base.cc:1034] running on GCE node
I20260810 14:38:14.909765 5372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:14.910470 5372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:14.911628 5372 hybrid_clock.cc:648] HybridClock initialized: now 1786372694911594 us; error 51 us; skew 500 ppm
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:14.914511 5372 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:14.915469 5372 webserver.cc:460] Webserver started at http://127.21.43.67:45403/ using document root <none> and password file <none>
I20260810 14:38:14.916016 5372 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:14.916090 5372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:14.920014 5372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:14.922181 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.923846 5372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:14.924002 5372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "d7bd93813fe745949c7ae930715c0105"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "068afb18640841ba6460f0e9983aae63"
server_key_iv: "f32aafaaf87e57b10679155df12e6e1d"
server_key_version: "encryptionkey@0"
I20260810 14:38:14.924785 5372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:14.967190 5372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:14.972108 5372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:14.972355 5372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:14.973261 5372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:14.974551 5372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:14.974617 5372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.974697 5372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:14.974731 5372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:14.984031 5372 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:38011
I20260810 14:38:14.984054 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:38011 every 8 connection(s)
I20260810 14:38:14.985092 5372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:14.993870 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5372
Aug 10 14:38:14 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372694, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:14.999389 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.986998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:43655 (local address 127.21.43.126:40161)
0810 14:38:14.987173 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.987178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.988077 (+ 899us) server_negotiation.cc:407] Connection header received
0810 14:38:14.989486 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.989490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.989540 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.989623 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:14.990805 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.991393 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.992493 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.992701 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.992779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.995494 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:14.995497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:14.995499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:14.995522 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:14.997241 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.997737 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.997741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.997743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.997841 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:14.998164 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:14.998168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:14.998170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:14.998377 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:14.998482 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:14.998988 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 14:38:14.999205 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:38:15.000277 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.987456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:43655)
0810 14:38:14.987925 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:14.987967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:14.989282 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:14.989788 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:14.989794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:14.990187 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:14.990672 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.990686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.991506 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.991508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.992390 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:14.992400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.992840 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.992842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:14.992982 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:14.993566 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:14.993583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:14.995328 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:14.997357 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.997362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.997377 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.997641 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.997935 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:14.997937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:14.997938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:14.998070 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:14.998483 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:14.998488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:14.998735 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 14:38:14.998941 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:38:14.999333 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260810 14:38:15.001605 5515 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40161
I20260810 14:38:15.001858 5515 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:15.002391 5515 heartbeater.cc:511] Master 127.21.43.126:40161 requested a full tablet report, sending...
I20260810 14:38:15.003551 5006 ts_manager.cc:194] Registered new tserver with Master: d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
I20260810 14:38:15.004136 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:14.994607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.21.43.126:40161)
0810 14:38:14.994894 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:14.994899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:14.995559 (+ 660us) server_negotiation.cc:407] Connection header received
0810 14:38:14.995658 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:14.995661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:14.995713 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:14.995824 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:14.996731 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.997519 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.998668 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:14.998904 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:14.999005 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.000740 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:15.000744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:15.000747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:15.000773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:15.002385 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:15.002820 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:15.002823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:15.002825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:15.002913 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:15.003249 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:15.003253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:15.003255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:15.003439 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:15.003674 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.003823 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.003956 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":62,"threads_started":1}
I20260810 14:38:15.004263 5006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:43655
I20260810 14:38:15.005978 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:15.010411 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.006583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.21.43.126:40161)
0810 14:38:15.006704 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.006708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.006925 (+ 217us) server_negotiation.cc:407] Connection header received
0810 14:38:15.007099 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.007102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.007266 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.007368 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:15.007644 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.008150 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.009004 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.009249 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.009360 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.009657 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:15.010018 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:15.010161 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.010220 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.010279 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:38:15.011021 5005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:38:15.014467 5005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:38:15.022753 5143 tablet_service.cc:1467] Processing CreateTablet for tablet e6e54ef255a1414383c1a9feea66e57f (DEFAULT_TABLE table=test-table [id=73dc23cea11b4843bb97925df88a7180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:15.023515 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e54ef255a1414383c1a9feea66e57f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:15.026268 5096 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap starting.
I20260810 14:38:15.027276 5096 tablet_bootstrap.cc:654] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:15.027982 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.022036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.21.43.67:38011)
0810 14:38:15.022363 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.022382 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.022400 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:38:15.022463 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.022467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.022636 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.022803 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:15.023110 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.024465 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.026291 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.027337 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.027487 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.027651 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.027756 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.027851 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":140,"threads_started":1}
I20260810 14:38:15.027967 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.021714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38011 (local address 127.0.0.1:53642)
0810 14:38:15.021971 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.021997 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:15.022083 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:15.022784 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:15.022787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:15.022807 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:15.022980 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.022994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.024595 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.024600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.026134 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.026160 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.027491 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.027495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.027587 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.027643 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 14:38:15.027725 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:38:15.027812 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":176,"spinlock_wait_cycles":7552,"thread_start_us":110,"threads_started":1}
I20260810 14:38:15.028501 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.021503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:40739 (local address 127.0.0.1:57496)
0810 14:38:15.021713 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.021730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:15.021817 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:15.022838 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:15.022842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:15.022870 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:15.022990 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.022997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.024635 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.024638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.026139 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.026148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.027674 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.027680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.027787 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.027823 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:38:15.027881 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 14:38:15.027961 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":119,"spinlock_wait_cycles":768}
I20260810 14:38:15.028548 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.021841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.21.43.66:40739)
0810 14:38:15.022355 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.022369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.022395 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:38:15.022473 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.022478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.022666 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.023195 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:15.023363 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.024466 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.026480 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.027494 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.028054 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.028232 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.028285 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.028376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":167,"threads_started":1}
I20260810 14:38:15.029564 5096 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: No bootstrap required, opened a new log
I20260810 14:38:15.029675 5096 ts_tablet_manager.cc:1397] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:38:15.030200 5096 raft_consensus.cc:348] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.030331 5096 raft_consensus.cc:374] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:15.030380 5096 raft_consensus.cc:729] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d43ab2307cc4d91b14971a1ed414fea, State: Initialized, Role: FOLLOWER
I20260810 14:38:15.030505 5096 consensus_queue.cc:260] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.030782 5096 ts_tablet_manager.cc:1428] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:38:15.031410 5443 tablet_service.cc:1467] Processing CreateTablet for tablet e6e54ef255a1414383c1a9feea66e57f (DEFAULT_TABLE table=test-table [id=73dc23cea11b4843bb97925df88a7180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:15.031605 5294 tablet_service.cc:1467] Processing CreateTablet for tablet e6e54ef255a1414383c1a9feea66e57f (DEFAULT_TABLE table=test-table [id=73dc23cea11b4843bb97925df88a7180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:15.032407 5443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e54ef255a1414383c1a9feea66e57f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:15.032655 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e54ef255a1414383c1a9feea66e57f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:15.039609 5531 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Bootstrap starting.
I20260810 14:38:15.040143 5532 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Bootstrap starting.
I20260810 14:38:15.041888 5532 tablet_bootstrap.cc:654] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:15.042330 5531 tablet_bootstrap.cc:654] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:15.043052 5532 log.cc:824] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:15.043052 5531 log.cc:824] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:15.045174 5531 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: No bootstrap required, opened a new log
I20260810 14:38:15.045205 5532 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: No bootstrap required, opened a new log
I20260810 14:38:15.045387 5531 ts_tablet_manager.cc:1397] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 14:38:15.045392 5532 ts_tablet_manager.cc:1397] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260810 14:38:15.047379 5217 tablet.cc:2369] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:38:15.048478 5532 raft_consensus.cc:348] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.048532 5531 raft_consensus.cc:348] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.048730 5532 raft_consensus.cc:374] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:15.048725 5531 raft_consensus.cc:374] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:15.048799 5531 raft_consensus.cc:729] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4408bbf3c30414c8a56d849a83f3b7d, State: Initialized, Role: FOLLOWER
I20260810 14:38:15.048811 5532 raft_consensus.cc:729] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bd93813fe745949c7ae930715c0105, State: Initialized, Role: FOLLOWER
I20260810 14:38:15.049252 5531 consensus_queue.cc:260] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.049252 5532 consensus_queue.cc:260] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.049932 5515 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
I20260810 14:38:15.049932 5369 heartbeater.cc:503] Master 127.21.43.126:40161 was elected leader, sending a full tablet report...
I20260810 14:38:15.050307 5531 ts_tablet_manager.cc:1428] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260810 14:38:15.051345 5532 ts_tablet_manager.cc:1428] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:38:15.071596 5536 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:15.071854 5536 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.073100 5536 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
I20260810 14:38:15.077589 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.073536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:34543 (local address 127.21.43.66:40739)
0810 14:38:15.073689 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.073693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.073710 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:38:15.073830 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.073833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.073880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.073967 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:15.074295 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.074982 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.076352 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.077005 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.077136 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.077195 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.077392 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.077461 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:38:15.077590 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.073406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:40739 (local address 127.21.43.67:34543)
0810 14:38:15.073560 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.073579 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:15.073689 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:15.073980 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:15.073983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:15.074003 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:15.074137 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.074144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.075112 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.075115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.076239 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.076252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.077142 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.077145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.077254 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.077279 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:38:15.077375 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:38:15.077448 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":5}
I20260810 14:38:15.077834 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.073523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:38569 (local address 127.21.43.65:42273)
0810 14:38:15.073652 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.073656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.074200 (+ 544us) server_negotiation.cc:407] Connection header received
0810 14:38:15.074315 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.074317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.074371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.074445 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:15.074754 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.075382 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.076603 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.077304 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.077434 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.077582 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.077656 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.077722 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:38:15.077855 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.073406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.21.43.67:38569)
0810 14:38:15.074101 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.074114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:15.074205 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:15.074480 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:15.074483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:15.074514 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:15.074638 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.074645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.075488 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.075491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.076476 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:15.076485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.077429 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.077432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:15.077537 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.077559 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:38:15.077634 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:38:15.077701 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":319,"thread_start_us":153,"threads_started":1}
I20260810 14:38:15.078339 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e54ef255a1414383c1a9feea66e57f" candidate_uuid: "d7bd93813fe745949c7ae930715c0105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" is_pre_election: true
I20260810 14:38:15.078478 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e54ef255a1414383c1a9feea66e57f" candidate_uuid: "d7bd93813fe745949c7ae930715c0105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d43ab2307cc4d91b14971a1ed414fea" is_pre_election: true
I20260810 14:38:15.078715 5318 raft_consensus.cc:2454] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7bd93813fe745949c7ae930715c0105 in term 0.
I20260810 14:38:15.078794 5163 raft_consensus.cc:2454] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7bd93813fe745949c7ae930715c0105 in term 0.
I20260810 14:38:15.079267 5389 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105; no voters:
I20260810 14:38:15.079582 5536 raft_consensus.cc:2781] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:15.079684 5536 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:15.079747 5536 raft_consensus.cc:3037] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:15.080833 5536 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.081233 5536 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [CANDIDATE]: Term 1 election: Requested vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273)
I20260810 14:38:15.081611 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e54ef255a1414383c1a9feea66e57f" candidate_uuid: "d7bd93813fe745949c7ae930715c0105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d43ab2307cc4d91b14971a1ed414fea"
I20260810 14:38:15.081632 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e54ef255a1414383c1a9feea66e57f" candidate_uuid: "d7bd93813fe745949c7ae930715c0105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
I20260810 14:38:15.081740 5163 raft_consensus.cc:3037] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:15.081773 5318 raft_consensus.cc:3037] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:15.082799 5163 raft_consensus.cc:2454] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7bd93813fe745949c7ae930715c0105 in term 1.
I20260810 14:38:15.082808 5318 raft_consensus.cc:2454] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7bd93813fe745949c7ae930715c0105 in term 1.
I20260810 14:38:15.083106 5389 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105; no voters:
I20260810 14:38:15.083334 5536 raft_consensus.cc:2781] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:15.083556 5536 raft_consensus.cc:686] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 1 LEADER]: Becoming Leader. State: Replica: d7bd93813fe745949c7ae930715c0105, State: Running, Role: LEADER
I20260810 14:38:15.083914 5536 consensus_queue.cc:237] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.087356 5005 catalog_manager.cc:5495] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7bd93813fe745949c7ae930715c0105 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "d7bd93813fe745949c7ae930715c0105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } health_report { overall_health: UNKNOWN } } }
I20260810 14:38:15.098371 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.094719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.21.43.67:38011)
0810 14:38:15.094850 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:15.094854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:15.094868 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:38:15.095005 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:15.095008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:15.095067 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:15.095213 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:15.095463 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.095965 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.096841 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:15.097064 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:15.097175 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:15.097492 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:15.097927 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:15.098065 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:15.098127 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 14:38:15.098186 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:38:15.104979 5163 raft_consensus.cc:1261] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Refusing update from remote peer d7bd93813fe745949c7ae930715c0105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:15.105028 5318 raft_consensus.cc:1261] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Refusing update from remote peer d7bd93813fe745949c7ae930715c0105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:15.105769 5536 consensus_queue.cc:1035] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:15.106055 5539 consensus_queue.cc:1035] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:15.115429 5544 mvcc.cc:205] Tried to move back new op lower bound from 7316982559141687296 to 7316982559066378240. Current Snapshot: MvccSnapshot[applied={T|T < 7316982559141687296}]
I20260810 14:38:15.122195 5006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:15.122370 5006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:15.124991 5006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Sending DeleteTablet for 3 replicas of tablet e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:15.125938 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5051
I20260810 14:38:15.125793 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e54ef255a1414383c1a9feea66e57f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:15 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57496
I20260810 14:38:15.125841 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e54ef255a1414383c1a9feea66e57f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:15 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:50978
I20260810 14:38:15.125962 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e54ef255a1414383c1a9feea66e57f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:15 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:53642
I20260810 14:38:15.130677 5555 tablet_replica.cc:330] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: stopping tablet replica
I20260810 14:38:15.131119 5555 raft_consensus.cc:2229] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:15.131575 5555 raft_consensus.cc:2258] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:15.132033 5553 tablet_replica.cc:330] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: stopping tablet replica
I20260810 14:38:15.132350 5553 raft_consensus.cc:2229] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:38:15.132812 5553 raft_consensus.cc:2258] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:15.133571 5555 ts_tablet_manager.cc:1905] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:15.134431 5553 ts_tablet_manager.cc:1905] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:15.136731 5555 ts_tablet_manager.cc:1918] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:15.136886 5555 log.cc:1196] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:15.137270 5555 ts_tablet_manager.cc:1939] T e6e54ef255a1414383c1a9feea66e57f P a4408bbf3c30414c8a56d849a83f3b7d: Deleting consensus metadata
I20260810 14:38:15.137809 5553 ts_tablet_manager.cc:1918] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:15.137992 5553 log.cc:1196] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:15.138429 5553 ts_tablet_manager.cc:1939] T e6e54ef255a1414383c1a9feea66e57f P d7bd93813fe745949c7ae930715c0105: Deleting consensus metadata
I20260810 14:38:15.138957 4990 catalog_manager.cc:4841] TS a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): tablet e6e54ef255a1414383c1a9feea66e57f (table test-table [id=73dc23cea11b4843bb97925df88a7180]) successfully deleted
W20260810 14:38:15.139333 4993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:42273 (error 108)
I20260810 14:38:15.139844 4993 catalog_manager.cc:4841] TS d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): tablet e6e54ef255a1414383c1a9feea66e57f (table test-table [id=73dc23cea11b4843bb97925df88a7180]) successfully deleted
I20260810 14:38:15.141099 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5221
I20260810 14:38:15.141417 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.141116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:42273 (local address 127.0.0.1:50982)
0810 14:38:15.141270 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.141336 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 14:38:15.144712 4993 catalog_manager.cc:4584] TS 0d43ab2307cc4d91b14971a1ed414fea (127.21.43.65:42273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6e54ef255a1414383c1a9feea66e57f: Network error: Client connection negotiation failed: client connection to 127.21.43.65:42273: connect: Connection refused (error 111)
I20260810 14:38:15.152242 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5372
I20260810 14:38:15.160137 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 4973
I20260810 14:38:15.169759 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--webserver_interface=127.21.43.126
--webserver_port=38665
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:15.286749 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:15.287212 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:15.287299 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:15.291848 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:15.291957 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:15.291993 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:15.292034 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:15.292069 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:15.297761 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=38665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:15.298991 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:15.300112 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:15.309345 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.309257 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.309386 5557 server_base.cc:1034] running on GCE node
W20260810 14:38:15.309571 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.310225 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:15.311414 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:15.312568 5557 hybrid_clock.cc:648] HybridClock initialized: now 1786372695312548 us; error 33 us; skew 500 ppm
Aug 10 14:38:15 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:15.315542 5557 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:15.316524 5557 webserver.cc:460] Webserver started at http://127.21.43.126:38665/ using document root <none> and password file <none>
I20260810 14:38:15.317143 5557 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:15.317212 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:15.321195 5557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:15.323354 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.324611 5557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:38:15.324755 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "4ab4f40bf5594ca7a3f1ce3272180199"
format_stamp: "Formatted at 2026-08-10 14:38:12 on dist-test-slave-chsk"
server_key: "b039512821845cb217313731562ac17a"
server_key_iv: "2125ce841e129ce1afd0638be52a8d71"
server_key_version: "encryptionkey@0"
I20260810 14:38:15.325384 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:15.353492 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:15.357436 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:15.357685 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:15.365059 5557 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40161
I20260810 14:38:15.365363 5631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40161 every 8 connection(s)
I20260810 14:38:15.366242 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:15.367759 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5557
I20260810 14:38:15.368251 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:42273
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=37697
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:15.371481 5633 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:38:15.373279 5633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap starting.
I20260810 14:38:15.391283 5633 log.cc:824] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:15.402896 5633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:15.403415 5633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199: Bootstrap complete.
I20260810 14:38:15.407606 5633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:15.408043 5633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Initialized, Role: FOLLOWER
I20260810 14:38:15.408535 5633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:15.408684 5633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:15.408776 5633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:15.408896 5633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:38:15.410801 5633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:15.411192 5633 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab4f40bf5594ca7a3f1ce3272180199; no voters:
I20260810 14:38:15.411594 5633 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:38:15.411679 5637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:38:15.411871 5637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [term 3 LEADER]: Becoming Leader. State: Replica: 4ab4f40bf5594ca7a3f1ce3272180199, State: Running, Role: LEADER
I20260810 14:38:15.412170 5637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } }
I20260810 14:38:15.412879 5633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:15.413652 5639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:15.414121 5639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:15.414352 5640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab4f40bf5594ca7a3f1ce3272180199. Latest consensus state: current_term: 3 leader_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab4f40bf5594ca7a3f1ce3272180199" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40161 } } }
I20260810 14:38:15.414644 5640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab4f40bf5594ca7a3f1ce3272180199 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:15.415117 5652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:15.420486 5652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:15.421185 5652 catalog_manager.cc:1259] Loaded cluster ID: f18227f4d2e4401aada656440c05463f
I20260810 14:38:15.421252 5652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 14:38:15.422415 5652 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc58a3b89ad google::LogMessage::Fail() at ??:0
@ 0x7fc58a3bb88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc58a3b84de google::LogMessage::Flush() at ??:0
@ 0x7fc58a3b9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc594cf7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc594cf8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc594cf5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc594d39640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc58c60a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc58c60ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc58c60c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc58c5fb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc58dc3f6db start_thread at ??:0
@ 0x7fc588eb661f clone at ??:0
W20260810 14:38:15.515383 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:15.515663 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:15.515745 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:15.519925 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:15.519999 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:15.520061 5635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:15.525682 5635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:42273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=37697
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:15.526943 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:15.528077 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:15.538242 5680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.538223 5682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.538275 5679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.538465 5635 server_base.cc:1034] running on GCE node
I20260810 14:38:15.539074 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:15.539979 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:15.541211 5635 hybrid_clock.cc:648] HybridClock initialized: now 1786372695541161 us; error 51 us; skew 500 ppm
Aug 10 14:38:15 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:15.544875 5635 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:15.546191 5635 webserver.cc:460] Webserver started at http://127.21.43.65:37697/ using document root <none> and password file <none>
I20260810 14:38:15.546952 5635 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:15.547055 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:15.552611 5635 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:38:15.555626 5690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.557334 5635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:15.557482 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "0d43ab2307cc4d91b14971a1ed414fea"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "da408b11dac915fac0bd53fe827052cc"
server_key_iv: "88094459bc33c722665298d5bfb1d162"
server_key_version: "encryptionkey@0"
I20260810 14:38:15.558254 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:15.574352 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:15.578452 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:15.578686 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:15.579397 5635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:15.580614 5697 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:38:15.586539 5635 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:38:15.586664 5635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 14:38:15.586880 5635 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:38:15.588759 5635 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:38:15.588840 5635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 14:38:15.589169 5697 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap starting.
I20260810 14:38:15.600020 5635 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:42273
I20260810 14:38:15.600404 5813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:42273 every 8 connection(s)
I20260810 14:38:15.601323 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:15.607359 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5635
I20260810 14:38:15.607801 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:40739
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=35707
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:15.614195 5697 log.cc:824] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:15.621716 5697 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:15.622181 5697 tablet_bootstrap.cc:492] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Bootstrap complete.
I20260810 14:38:15.623508 5697 ts_tablet_manager.cc:1397] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.004s
I20260810 14:38:15.626302 5697 raft_consensus.cc:348] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.626796 5697 raft_consensus.cc:729] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d43ab2307cc4d91b14971a1ed414fea, State: Initialized, Role: FOLLOWER
I20260810 14:38:15.627360 5697 consensus_queue.cc:260] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:15.628319 5697 ts_tablet_manager.cc:1428] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260810 14:38:15.672245 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:38:15.603553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:45003)
0810 14:38:15.604115 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.604173 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:15.605355 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:15.671435 (+ 66080us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":164,"threads_started":1}
W20260810 14:38:15.672902 5814 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:40161 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:40161: Client connection negotiation failed: client connection to 127.21.43.126:40161: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:38:15.673720 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.673429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:60675)
0810 14:38:15.673581 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.673658 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:38:15.674451 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.674230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:35719)
0810 14:38:15.674339 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.674388 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:38:15.674715 5814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:38:15.748731 5817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:15.748997 5817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:15.749073 5817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:15.753192 5817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:15.753257 5817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:15.753364 5817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:15.758921 5817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:40739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=35707
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:15.760044 5817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:15.761076 5817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:38:15.771121 5817 server_base.cc:1034] running on GCE node
W20260810 14:38:15.771092 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.771076 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.771071 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.771791 5817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:15.772507 5817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:15.773702 5817 hybrid_clock.cc:648] HybridClock initialized: now 1786372695773661 us; error 54 us; skew 500 ppm
Aug 10 14:38:15 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:15.776767 5817 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:15.777805 5817 webserver.cc:460] Webserver started at http://127.21.43.66:35707/ using document root <none> and password file <none>
I20260810 14:38:15.778414 5817 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:15.778514 5817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:15.782601 5817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:15.784999 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.786469 5817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:15.786664 5817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "a4408bbf3c30414c8a56d849a83f3b7d"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "70167493f55ca066de12b63066ee1578"
server_key_iv: "d88f88b442ff0915395e68c61a07916b"
server_key_version: "encryptionkey@0"
I20260810 14:38:15.787444 5817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:15.803061 5817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:15.806156 5817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:15.806404 5817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:15.807039 5817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:15.807925 5817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:15.807974 5817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.808014 5817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:15.808029 5817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.817556 5817 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:40739
I20260810 14:38:15.817646 5958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:40739 every 8 connection(s)
I20260810 14:38:15.818625 5817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:15.822825 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.820921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:42669)
0810 14:38:15.821454 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.821573 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":129,"threads_started":1}
W20260810 14:38:15.823359 5959 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:40161 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:40161: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
I20260810 14:38:15.823973 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.823761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:60755)
0810 14:38:15.823870 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.823919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:38:15.824633 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:15.824444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:52341)
0810 14:38:15.824543 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:15.824584 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:38:15.824842 5959 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:38:15.827721 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5817
I20260810 14:38:15.828174 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:38011
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:40161
--webserver_port=45403
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:15.953397 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:15.953704 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:15.953751 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:15.957906 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:15.957983 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:15.958043 5962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:15.963548 5962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:38011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=45403
--tserver_master_addrs=127.21.43.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:15.964697 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:15.965795 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:15.975698 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:15.975709 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.975883 5962 server_base.cc:1034] running on GCE node
W20260810 14:38:15.975698 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:15.976596 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:15.977301 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:15.978495 5962 hybrid_clock.cc:648] HybridClock initialized: now 1786372695978476 us; error 34 us; skew 500 ppm
Aug 10 14:38:15 dist-test-slave-chsk krb5kdc[4370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:15.981463 5962 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:15.982442 5962 webserver.cc:460] Webserver started at http://127.21.43.67:45403/ using document root <none> and password file <none>
I20260810 14:38:15.983024 5962 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:15.983127 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:15.988180 5962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:38:15.990417 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:15.992174 5962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:38:15.992331 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "d7bd93813fe745949c7ae930715c0105"
format_stamp: "Formatted at 2026-08-10 14:38:13 on dist-test-slave-chsk"
server_key: "068afb18640841ba6460f0e9983aae63"
server_key_iv: "f32aafaaf87e57b10679155df12e6e1d"
server_key_version: "encryptionkey@0"
I20260810 14:38:15.992931 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:16.010733 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:16.014174 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:16.014401 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:16.015043 5962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:16.015971 5962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:16.016052 5962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:16.016120 5962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:16.016165 5962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:16.025353 5962 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:38011
I20260810 14:38:16.025388 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:38011 every 8 connection(s)
I20260810 14:38:16.026512 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:16.030568 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:16.028770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:42641)
0810 14:38:16.029249 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:16.029367 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
W20260810 14:38:16.031065 6103 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:40161 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:40161: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
I20260810 14:38:16.031708 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:16.031490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:44441)
0810 14:38:16.031599 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:16.031648 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 14:38:16.032500 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:16.032307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:57003)
0810 14:38:16.032410 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:16.032451 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:38:16.032713 6103 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:38:16.036785 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 5962
I20260810 14:38:16.676115 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:16.675565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:34025)
0810 14:38:16.675891 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:16.675969 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260810 14:38:16.825959 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:16.825497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:58249)
0810 14:38:16.825789 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:16.825846 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260810 14:38:17.033831 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.033363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:46079)
0810 14:38:17.033658 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:17.033721 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260810 14:38:17.062808 6112 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:17.063068 6112 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:17.064399 6112 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
Aug 10 14:38:17 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.66@KRBTEST.COM
Aug 10 14:38:17 dist-test-slave-chsk krb5kdc[4370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372695, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.67@KRBTEST.COM
I20260810 14:38:17.083525 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.064661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:40739 (local address 127.21.43.65:49113)
0810 14:38:17.064843 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:17.064857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:17.064947 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:17.067138 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:17.067170 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:17.067818 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:17.068647 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:17.068668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.070585 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.070588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:17.072266 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:17.072279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.073333 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.073337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:17.073709 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:17.074526 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:17.074553 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:17.076773 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:17.079954 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:17.079961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:17.079978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:17.080513 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:17.081004 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:17.081007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:17.081009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:17.081256 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:17.082144 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:17.082151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:17.082532 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 14:38:17.082772 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 14:38:17.083280 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":575,"spinlock_wait_cycles":61056}
I20260810 14:38:17.083554 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.064829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:49113 (local address 127.21.43.66:40739)
0810 14:38:17.065233 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:17.065256 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:17.065303 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:38:17.066239 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:17.066251 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:17.066704 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:17.067042 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:17.068849 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.070406 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.072414 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.073187 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.073374 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:17.077090 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:17.077099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:17.077117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:17.077170 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:17.079799 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:17.080700 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:17.080708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:17.080714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:17.080890 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:17.081382 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:17.081385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:17.081387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:17.082024 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:17.082161 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:17.082987 (+ 826us) server_negotiation.cc:299] Negotiation successful
0810 14:38:17.083350 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260810 14:38:17.083554 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.064817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:44803 (local address 127.21.43.67:38011)
0810 14:38:17.065186 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:17.065216 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:17.065521 (+ 305us) server_negotiation.cc:407] Connection header received
0810 14:38:17.066329 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:17.066336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:17.066704 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:17.067042 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:17.068849 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.070406 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.072414 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.073176 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.073374 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:17.077171 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:17.077177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:17.077187 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:17.077223 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:17.079799 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:17.080700 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:17.080708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:17.080714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:17.080890 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:17.081382 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:17.081385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:17.081388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:17.082000 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:17.082161 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:17.082985 (+ 824us) server_negotiation.cc:299] Negotiation successful
0810 14:38:17.083349 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260810 14:38:17.083525 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.064665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:38011 (local address 127.21.43.65:44803)
0810 14:38:17.065414 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:17.065426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:17.065518 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:17.067158 (+ 1640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:17.067165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:17.067833 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:17.068647 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:17.068668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.070584 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.070588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:17.072265 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:17.072279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:17.073333 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:17.073337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:17.073718 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:17.074528 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:17.074548 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:17.076981 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:17.079954 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:17.079961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:17.079978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:17.080513 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:17.081004 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:17.081007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:17.081009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:17.081250 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:17.082144 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:17.082151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:17.082532 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 14:38:17.082772 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 14:38:17.083336 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":1006,"spinlock_wait_cycles":36224,"thread_start_us":57,"threads_started":1}
W20260810 14:38:17.086822 5694 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
W20260810 14:38:17.087049 5691 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:17.087261 5691 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea; no voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105
I20260810 14:38:17.087597 6112 raft_consensus.cc:2726] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:38:17.677330 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.676907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:48705)
0810 14:38:17.677169 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:17.677230 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260810 14:38:17.827273 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:17.826745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:47477)
0810 14:38:17.827058 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:17.827120 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":86,"threads_started":1}
I20260810 14:38:18.035087 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:18.034668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:54209)
0810 14:38:18.034933 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:18.034992 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260810 14:38:18.632726 6121 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:18.632877 6121 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:18.633440 6121 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
W20260810 14:38:18.634398 5691 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
W20260810 14:38:18.634545 5694 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:18.634625 5694 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea; no voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105
I20260810 14:38:18.634758 6121 raft_consensus.cc:2726] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:38:18.678629 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:18.678161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:38801)
0810 14:38:18.678456 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:18.678521 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260810 14:38:18.828589 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:18.828149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:36393)
0810 14:38:18.828428 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:18.828489 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260810 14:38:19.036473 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:19.035997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:49397)
0810 14:38:19.036307 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:19.036370 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260810 14:38:19.679899 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:19.679466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:43105)
0810 14:38:19.679738 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:19.679798 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260810 14:38:19.830313 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:19.829796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:44285)
0810 14:38:19.830114 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:19.830202 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260810 14:38:20.037858 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:20.037355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:47617)
0810 14:38:20.037689 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:20.037754 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260810 14:38:20.681397 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:20.680899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:37193)
0810 14:38:20.681234 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:20.681298 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":98,"threads_started":1}
I20260810 14:38:20.831830 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:20.831366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:41013)
0810 14:38:20.831662 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:20.831727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260810 14:38:21.039345 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:21.038813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:54559)
0810 14:38:21.039170 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:21.039238 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260810 14:38:21.682659 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:21.682207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:54367)
0810 14:38:21.682496 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:21.682555 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260810 14:38:21.778002 6133 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:21.778189 6133 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:21.778779 6133 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
W20260810 14:38:21.779699 5691 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
W20260810 14:38:21.779848 5694 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:21.779947 5694 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea; no voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105
I20260810 14:38:21.780153 6133 raft_consensus.cc:2726] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:38:21.833326 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:21.832790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:50961)
0810 14:38:21.833130 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:21.833200 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":89,"threads_started":1}
I20260810 14:38:22.040675 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:22.040230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:34101)
0810 14:38:22.040520 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:22.040577 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260810 14:38:22.684159 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:22.683655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:58553)
0810 14:38:22.683968 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:22.684045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260810 14:38:22.834748 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:22.834279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:42623)
0810 14:38:22.834587 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:22.834647 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260810 14:38:23.042045 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:23.041516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:56427)
0810 14:38:23.041862 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:23.041924 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":86,"threads_started":1}
I20260810 14:38:23.685559 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:23.685037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:60421)
0810 14:38:23.685370 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:23.685434 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260810 14:38:23.836143 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:23.835655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:51515)
0810 14:38:23.835980 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:23.836042 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260810 14:38:24.043402 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:24.042875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:52519)
0810 14:38:24.043234 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:24.043299 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260810 14:38:24.686872 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:24.686399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:40049)
0810 14:38:24.686705 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:24.686764 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
I20260810 14:38:24.837524 6143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:24.837043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:33007)
0810 14:38:24.837359 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:24.837424 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260810 14:38:25.044682 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:25.044217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:45715)
0810 14:38:25.044502 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:25.044564 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260810 14:38:25.689148 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:25.688561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:35293)
0810 14:38:25.688960 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:25.689038 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260810 14:38:25.714767 6146 raft_consensus.cc:482] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:25.714943 6146 raft_consensus.cc:504] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4408bbf3c30414c8a56d849a83f3b7d" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 40739 } } peers { permanent_uuid: "d7bd93813fe745949c7ae930715c0105" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 38011 } } peers { permanent_uuid: "0d43ab2307cc4d91b14971a1ed414fea" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 42273 } }
I20260810 14:38:25.715547 6146 leader_election.cc:288] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739), d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011)
W20260810 14:38:25.716476 5694 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7bd93813fe745949c7ae930715c0105 (127.21.43.67:38011): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
W20260810 14:38:25.716642 5691 leader_election.cc:341] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4408bbf3c30414c8a56d849a83f3b7d (127.21.43.66:40739): Not found: Tablet not found: e6e54ef255a1414383c1a9feea66e57f
I20260810 14:38:25.716737 5691 leader_election.cc:302] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d43ab2307cc4d91b14971a1ed414fea; no voters: a4408bbf3c30414c8a56d849a83f3b7d, d7bd93813fe745949c7ae930715c0105
I20260810 14:38:25.716938 6146 raft_consensus.cc:2726] T e6e54ef255a1414383c1a9feea66e57f P 0d43ab2307cc4d91b14971a1ed414fea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:38:25.838910 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:25.838418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:34893)
0810 14:38:25.838759 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:25.838812 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260810 14:38:26.046161 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:26.045669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:48495)
0810 14:38:26.045970 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:26.046039 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260810 14:38:26.691208 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:26.690168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:35767)
0810 14:38:26.690944 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:26.691032 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260810 14:38:26.840483 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:26.839921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:54309)
0810 14:38:26.840315 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:26.840378 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":88,"threads_started":1}
I20260810 14:38:27.047765 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:27.047110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:54239)
0810 14:38:27.047587 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:27.047656 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":96,"threads_started":1}
I20260810 14:38:27.692854 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:27.692388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:40073)
0810 14:38:27.692681 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:27.692748 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260810 14:38:27.841918 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:27.841416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:47859)
0810 14:38:27.841733 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:27.841812 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
I20260810 14:38:28.049144 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:28.048676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:35053)
0810 14:38:28.048957 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:28.049034 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260810 14:38:28.694207 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:28.693690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:37319)
0810 14:38:28.694031 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:28.694096 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
I20260810 14:38:28.843567 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:28.842966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:39133)
0810 14:38:28.843382 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:28.843453 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":120,"threads_started":1}
I20260810 14:38:29.051267 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:29.050363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:59315)
0810 14:38:29.050876 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:29.051033 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":160,"threads_started":1}
I20260810 14:38:29.695562 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:29.695077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:56501)
0810 14:38:29.695403 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:29.695463 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
I20260810 14:38:29.845078 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:29.844517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:56865)
0810 14:38:29.844865 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:29.844932 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260810 14:38:30.052738 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:30.052192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:43831)
0810 14:38:30.052565 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:30.052635 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260810 14:38:30.697283 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:30.696737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.65:35453)
0810 14:38:30.697085 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:30.697150 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260810 14:38:30.846474 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:30.846022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.66:49805)
0810 14:38:30.846314 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:30.846374 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260810 14:38:31.043344 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5635
I20260810 14:38:31.054138 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5817
I20260810 14:38:31.056581 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:31.053838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40161 (local address 127.21.43.67:34291)
0810 14:38:31.054146 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:31.054202 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:40161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260810 14:38:31.064630 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 5962
2026-08-10T14:38:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19530 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:38:31 dist-test-slave-chsk krb5kdc[6168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:38:31 dist-test-slave-chsk krb5kdc[6168](info): set up 2 sockets
Aug 10 14:38:31 dist-test-slave-chsk krb5kdc[6168](info): commencing operation
krb5kdc: starting...
W20260810 14:38:32.186112 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:38:32.322566 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:32.437400 6182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:32.437722 6182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:32.437808 6182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:32.441845 6182 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:38:32.441929 6182 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:38:32.441957 6182 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:38:32.442018 6182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:32.442058 6182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:32.442114 6182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:32.442153 6182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:32.442188 6182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:32.447589 6182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:32.448736 6182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:32.449810 6182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:38:32.458981 6182 server_base.cc:1034] running on GCE node
W20260810 14:38:32.459043 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:32.459018 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:32.459072 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:32.459930 6182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:32.461043 6182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:32.462215 6182 hybrid_clock.cc:648] HybridClock initialized: now 1786372712462196 us; error 31 us; skew 500 ppm
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:32.465184 6182 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:32.466120 6182 webserver.cc:460] Webserver started at http://127.21.43.126:37605/ using document root <none> and password file <none>
I20260810 14:38:32.466709 6182 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:32.466775 6182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:32.467001 6182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:32.469128 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "81db456f5d1b408db740a2444681e316"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "e4266b99d9b276926bf58ea8c14edb6d"
server_key_iv: "feab6b1b9aac7e1290d823c1f597ea7c"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.469775 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "81db456f5d1b408db740a2444681e316"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "e4266b99d9b276926bf58ea8c14edb6d"
server_key_iv: "feab6b1b9aac7e1290d823c1f597ea7c"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.473629 6182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:38:32.476094 6201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:32.477422 6182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:38:32.477591 6182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "81db456f5d1b408db740a2444681e316"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "e4266b99d9b276926bf58ea8c14edb6d"
server_key_iv: "feab6b1b9aac7e1290d823c1f597ea7c"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.477710 6182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:32.495908 6182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:32.499454 6182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:32.499665 6182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:32.506712 6182 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38751
I20260810 14:38:32.507027 6261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38751 every 8 connection(s)
I20260810 14:38:32.507759 6182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:32.510353 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6182
I20260810 14:38:32.510459 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:38:32.510802 21677 external_mini_cluster.cc:1428] Setting key ce0c41b3f3985cb841dfa482eb64f147
I20260810 14:38:32.510818 6262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:32.518707 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap starting.
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:32.521723 6262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:32.522785 6262 log.cc:824] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:32.525091 6262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: No bootstrap required, opened a new log
I20260810 14:38:32.526412 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:32.512337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.21.43.126:38751)
0810 14:38:32.512802 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:32.512823 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:32.512869 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:38:32.513781 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:32.513806 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:32.514273 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:32.514686 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:32.515481 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.516944 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.517769 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.518445 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.518611 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:32.521648 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:32.521657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:32.521678 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:32.521715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:32.523598 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:32.524021 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:32.524026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:32.524032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:32.524176 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:32.524436 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:32.524440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:32.524441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:32.524934 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:32.525124 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:32.525434 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 14:38:32.525729 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260810 14:38:32.529258 6262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:32.529492 6262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:32.529587 6262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db456f5d1b408db740a2444681e316, State: Initialized, Role: FOLLOWER
I20260810 14:38:32.530053 6262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:32.530210 6262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:32.530282 6262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:32.530386 6262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:32.531530 6262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:32.531896 6262 leader_election.cc:302] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db456f5d1b408db740a2444681e316; no voters:
I20260810 14:38:32.532270 6262 leader_election.cc:288] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:38:32.532434 6267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:32.532699 6267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 LEADER]: Becoming Leader. State: Replica: 81db456f5d1b408db740a2444681e316, State: Running, Role: LEADER
I20260810 14:38:32.532992 6267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:32.533509 6262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:32.534700 6269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db456f5d1b408db740a2444681e316. Latest consensus state: current_term: 1 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:38:32.534837 6269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:32.535095 6268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:38:32.535234 6268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:32.535742 6278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:32.538328 6278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:32.543761 6278 catalog_manager.cc:1347] Generated new cluster ID: cad90e6661374b96a44c403eaf7fa650
I20260810 14:38:32.543836 6278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:32.559393 6278 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:38:32.560393 6278 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:32.568076 6278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Generated new TSK 0
I20260810 14:38:32.569034 6278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:38:32.611387 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:32.729233 6292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:32.729545 6292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:32.729645 6292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:32.734012 6292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:32.734098 6292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:32.734194 6292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:32.739909 6292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:32.741086 6292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:32.742202 6292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:32.751873 6300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:32.751878 6298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:32.751931 6292 server_base.cc:1034] running on GCE node
W20260810 14:38:32.752177 6297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:32.752698 6292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:32.753322 6292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:32.754513 6292 hybrid_clock.cc:648] HybridClock initialized: now 1786372712754490 us; error 30 us; skew 500 ppm
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:32.757368 6292 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:32.758260 6292 webserver.cc:460] Webserver started at http://127.21.43.65:40757/ using document root <none> and password file <none>
I20260810 14:38:32.758847 6292 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:32.758910 6292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:32.759233 6292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:32.761353 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "eada159a0c404de5b69162d58221c6a0"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "5c24bcdcd48b8d44ec24b47768a4336d"
server_key_iv: "a84f12520d2482a319409ff2042bd561"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.761950 6292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "eada159a0c404de5b69162d58221c6a0"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "5c24bcdcd48b8d44ec24b47768a4336d"
server_key_iv: "a84f12520d2482a319409ff2042bd561"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.765743 6292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260810 14:38:32.768148 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:32.769413 6292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:38:32.769555 6292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "eada159a0c404de5b69162d58221c6a0"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "5c24bcdcd48b8d44ec24b47768a4336d"
server_key_iv: "a84f12520d2482a319409ff2042bd561"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.769675 6292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:32.791923 6292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:32.794973 6292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:32.795260 6292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:32.795917 6292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:32.796798 6292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:32.796873 6292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:32.796948 6292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:32.797000 6292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:32.806316 6292 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39755
I20260810 14:38:32.806341 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39755 every 8 connection(s)
I20260810 14:38:32.807366 6292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:32.809445 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6292
I20260810 14:38:32.809639 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:38:32.810005 21677 external_mini_cluster.cc:1428] Setting key 760e96f6fea1a76ec60e9e5d428e1947
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:38:32.822378 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:32.809303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:52651 (local address 127.21.43.126:38751)
0810 14:38:32.809430 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:32.809434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:32.810606 (+ 1172us) server_negotiation.cc:407] Connection header received
0810 14:38:32.812232 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:32.812236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:32.812290 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:32.812367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:32.813574 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.814127 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.815358 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.815614 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.815731 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:32.818586 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:32.818590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:32.818592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:32.818626 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:32.820138 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:32.820690 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:32.820694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:32.820696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:32.820780 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:32.821140 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:32.821144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:32.821145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:32.821363 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:32.821473 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:32.821997 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:38:32.822201 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:38:32.823336 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:32.809793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:52651)
0810 14:38:32.810419 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:32.810478 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:32.812006 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:32.812546 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:32.812552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:32.812924 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:32.813417 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:32.813430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.814260 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.814263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:32.815224 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:32.815235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:32.815793 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:32.815797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:32.815978 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:32.816632 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:32.816651 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:32.818408 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:32.820264 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:32.820271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:32.820287 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:32.820578 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:32.820879 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:32.820882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:32.820884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:32.821031 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:32.821474 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:32.821481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:32.821735 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 14:38:32.821928 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 14:38:32.822294 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":146,"threads_started":1}
I20260810 14:38:32.824703 6440 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:32.824961 6440 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:32.825474 6440 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:32.827211 6221 ts_manager.cc:194] Registered new tserver with Master: eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755)
I20260810 14:38:32.829030 6221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:52651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:38:32.842170 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:32.958082 6445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:32.958384 6445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:32.958428 6445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:32.962551 6445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:32.962625 6445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:32.962687 6445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:32.968047 6445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:32.969099 6445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:32.970096 6445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:32.979992 6450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:32.979995 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:32.979995 6451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:32.980379 6445 server_base.cc:1034] running on GCE node
I20260810 14:38:32.980898 6445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:32.981618 6445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:32.982812 6445 hybrid_clock.cc:648] HybridClock initialized: now 1786372712982798 us; error 29 us; skew 500 ppm
Aug 10 14:38:32 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:32.985687 6445 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:32.986613 6445 webserver.cc:460] Webserver started at http://127.21.43.66:44509/ using document root <none> and password file <none>
I20260810 14:38:32.987226 6445 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:32.987298 6445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:32.987468 6445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:32.989456 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "428884ca65c047929d3234cd7c596309"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "d6311add3a2a4a67081c369e67e56fcc"
server_key_iv: "bb14e5b0013f51ed8b01dd2865ad4b79"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.989979 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "428884ca65c047929d3234cd7c596309"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "d6311add3a2a4a67081c369e67e56fcc"
server_key_iv: "bb14e5b0013f51ed8b01dd2865ad4b79"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.994179 6445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:38:32.996716 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:32.998018 6445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:38:32.998186 6445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "428884ca65c047929d3234cd7c596309"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "d6311add3a2a4a67081c369e67e56fcc"
server_key_iv: "bb14e5b0013f51ed8b01dd2865ad4b79"
server_key_version: "encryptionkey@0"
I20260810 14:38:32.998323 6445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:33.016878 6445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:33.019928 6445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:33.020148 6445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:33.020771 6445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:33.021636 6445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:33.021685 6445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:33.021752 6445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:33.021791 6445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:33.031195 6445 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45861
I20260810 14:38:33.031234 6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45861 every 8 connection(s)
I20260810 14:38:33.032120 6445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:33.040045 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6445
I20260810 14:38:33.040194 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:38:33.040490 21677 external_mini_cluster.cc:1428] Setting key fc1b30f71000604d22361cb44dcf45e6
Aug 10 14:38:33 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:33.046566 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.034010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:39745 (local address 127.21.43.126:38751)
0810 14:38:33.034140 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.034145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.035116 (+ 971us) server_negotiation.cc:407] Connection header received
0810 14:38:33.036666 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.036669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.036722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.036806 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:33.037926 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.038458 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.039627 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.039927 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.040041 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.042961 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:33.042970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:33.042972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:33.042996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:33.044496 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.045003 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.045008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.045010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.045102 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.045443 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.045448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.045450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.045644 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:33.045754 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.046257 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.046389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:38:33.047544 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.034416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:39745)
0810 14:38:33.034954 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.035006 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.036461 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.036961 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.036967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.037332 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:33.037781 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.037794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.038564 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.038567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.039523 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.039533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.040094 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.040097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.040332 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.041149 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:33.041166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:33.042792 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:33.044618 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:33.044624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:33.044638 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:33.044895 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:33.045197 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:33.045200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:33.045202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:33.045344 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:33.045752 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:33.045759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:33.046016 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.046210 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.046532 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":153,"threads_started":1}
I20260810 14:38:33.048734 6587 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:33.048990 6587 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:33.049487 6587 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:33.050755 6221 ts_manager.cc:194] Registered new tserver with Master: 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861)
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:38:33.051715 6221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:39745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:38:33.074683 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:33.191431 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:33.191744 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:33.191821 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:33.195894 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:33.195992 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:33.196084 6592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:33.202224 6592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:33.203502 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:33.204602 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:33.214741 6599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:33.214741 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:33.214989 6592 server_base.cc:1034] running on GCE node
W20260810 14:38:33.214790 6598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:33.215463 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:33.216207 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:33.217427 6592 hybrid_clock.cc:648] HybridClock initialized: now 1786372713217391 us; error 49 us; skew 500 ppm
Aug 10 14:38:33 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:33.220518 6592 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:33.221542 6592 webserver.cc:460] Webserver started at http://127.21.43.67:33325/ using document root <none> and password file <none>
I20260810 14:38:33.222153 6592 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:33.222251 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:33.222493 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:38:33.224668 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "65e1c648988949fc9f1250f6492c01e4"
format_stamp: "Formatted at 2026-08-10 14:38:33 on dist-test-slave-chsk"
server_key: "0d905a8689e1c26c868d403fe778be1e"
server_key_iv: "e7e9415adb87600acd4faee8ada171ae"
server_key_version: "encryptionkey@0"
I20260810 14:38:33.225265 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "65e1c648988949fc9f1250f6492c01e4"
format_stamp: "Formatted at 2026-08-10 14:38:33 on dist-test-slave-chsk"
server_key: "0d905a8689e1c26c868d403fe778be1e"
server_key_iv: "e7e9415adb87600acd4faee8ada171ae"
server_key_version: "encryptionkey@0"
I20260810 14:38:33.229139 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:38:33.231720 6608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:33.232910 6592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:38:33.233095 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "65e1c648988949fc9f1250f6492c01e4"
format_stamp: "Formatted at 2026-08-10 14:38:33 on dist-test-slave-chsk"
server_key: "0d905a8689e1c26c868d403fe778be1e"
server_key_iv: "e7e9415adb87600acd4faee8ada171ae"
server_key_version: "encryptionkey@0"
I20260810 14:38:33.233224 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:33.247548 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:33.251017 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:33.251282 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:33.251948 6592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:33.252912 6592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:33.252985 6592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:33.253050 6592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:33.253100 6592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:33.261919 6592 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:36923
I20260810 14:38:33.261941 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:36923 every 8 connection(s)
I20260810 14:38:33.262912 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:33.272552 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6592
I20260810 14:38:33.272710 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:38:33.272997 21677 external_mini_cluster.cc:1428] Setting key 27ba70aca3cbe846aca76a15cd529434
Aug 10 14:38:33 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:33.277741 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.264872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:56473 (local address 127.21.43.126:38751)
0810 14:38:33.265024 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.265028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.265864 (+ 836us) server_negotiation.cc:407] Connection header received
0810 14:38:33.267339 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.267342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.267405 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.267511 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:33.268748 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.269297 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.270482 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.270709 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.270825 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.273798 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:33.273801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:33.273804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:33.273837 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:33.275684 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.276172 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.276174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.276176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.276261 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.276572 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.276574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.276575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.276768 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:33.276853 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.277486 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.277609 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:38:33.278723 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.265231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:56473)
0810 14:38:33.265700 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.265744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.267034 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.267703 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.267710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.268091 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:33.268588 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.268601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.269437 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.269440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.270336 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.270346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.270878 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.270882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.271057 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.271740 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:33.271757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:33.273610 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:33.275796 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:33.275801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:33.275814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:33.276079 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:33.276343 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:33.276346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:33.276347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:33.276488 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:33.276899 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:33.276905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:33.277216 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.277417 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.277729 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
I20260810 14:38:33.280010 6734 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:33.280287 6734 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:33.280885 6734 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:33.282159 6221 ts_manager.cc:194] Registered new tserver with Master: 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
I20260810 14:38:33.282728 6221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:56473
I20260810 14:38:33.288460 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:33.300745 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.291668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.21.43.126:38751)
0810 14:38:33.291818 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.291825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.292067 (+ 242us) server_negotiation.cc:407] Connection header received
0810 14:38:33.292294 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.292298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.292350 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.292445 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:33.293396 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.294134 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.295049 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.295362 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.295471 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.296908 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:33.296912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:33.296915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:33.296943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:33.298894 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.299397 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.299401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.299403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.299496 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:33.299778 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:33.299782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:33.299784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:33.300045 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:33.300196 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.300439 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.300565 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:38:33.304586 6221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:38:33.307775 6221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:38:33.331736 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.323806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45861 (local address 127.0.0.1:38274)
0810 14:38:33.324443 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.324501 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.324647 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.326063 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.326071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.326100 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:33.326274 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.326285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328561 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.328565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.329935 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.329946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.331071 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.331075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.331278 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.331332 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.331421 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.331551 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":144,"thread_start_us":100,"threads_started":1}
I20260810 14:38:33.332206 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.324009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.0.0.1:40392)
0810 14:38:33.325536 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.325548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.325671 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.326308 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.326313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.326336 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:33.326459 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.326468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328238 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.328243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.330030 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.330045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.331659 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.331664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.331804 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.331840 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.331921 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.332014 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"mutex_wait_us":2,"thread_start_us":146,"threads_started":1}
I20260810 14:38:33.332481 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.325268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36923 (local address 127.0.0.1:59328)
0810 14:38:33.326347 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.326360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.326444 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.327025 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.327028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.327044 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:33.327195 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.327203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328579 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.328581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.330761 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.330772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.332060 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.332066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.332179 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.332202 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.332269 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.332358 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":66,"thread_start_us":124,"threads_started":1}
I20260810 14:38:33.332847 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.324164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.21.43.65:39755)
0810 14:38:33.324557 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.324578 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.325663 (+ 1085us) server_negotiation.cc:407] Connection header received
0810 14:38:33.325786 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.325794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.326015 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.326168 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:33.326611 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328056 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.330197 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.331234 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.331343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.331538 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.332496 (+ 958us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.332638 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":80,"threads_started":1}
I20260810 14:38:33.332908 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.325458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.21.43.67:36923)
0810 14:38:33.326220 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.326231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.326524 (+ 293us) server_negotiation.cc:407] Connection header received
0810 14:38:33.326596 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.326600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.326759 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.326885 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:33.327332 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328444 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.330907 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.331721 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.331852 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.331999 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.332664 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.332768 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":171,"threads_started":1}
I20260810 14:38:33.334358 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.324789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.21.43.66:45861)
0810 14:38:33.325079 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.325509 (+ 430us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.325544 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:38:33.325648 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.325654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.325889 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.326057 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:33.327187 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.328387 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.330075 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.330941 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.332834 (+ 1893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.334040 (+ 1206us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.334096 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.334203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260810 14:38:33.336839 6511 tablet_service.cc:1467] Processing CreateTablet for tablet 9532c98c47fa4119b2d7d833664024c6 (DEFAULT_TABLE table=test-table [id=b83f8014b094417f83b2892d931f11b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:33.336727 6360 tablet_service.cc:1467] Processing CreateTablet for tablet 9532c98c47fa4119b2d7d833664024c6 (DEFAULT_TABLE table=test-table [id=b83f8014b094417f83b2892d931f11b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:33.337962 6511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9532c98c47fa4119b2d7d833664024c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:33.338063 6360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9532c98c47fa4119b2d7d833664024c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:33.341702 6662 tablet_service.cc:1467] Processing CreateTablet for tablet 9532c98c47fa4119b2d7d833664024c6 (DEFAULT_TABLE table=test-table [id=b83f8014b094417f83b2892d931f11b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:33.343003 6662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9532c98c47fa4119b2d7d833664024c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:33.344676 6751 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Bootstrap starting.
I20260810 14:38:33.348048 6751 tablet_bootstrap.cc:654] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:33.349049 6751 log.cc:824] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:33.351373 6751 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: No bootstrap required, opened a new log
I20260810 14:38:33.351671 6751 ts_tablet_manager.cc:1397] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:38:33.351818 6753 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Bootstrap starting.
I20260810 14:38:33.353034 6754 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Bootstrap starting.
I20260810 14:38:33.354707 6753 tablet_bootstrap.cc:654] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:33.355513 6754 tablet_bootstrap.cc:654] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:33.355749 6753 log.cc:824] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:33.356354 6754 log.cc:824] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:33.357379 6751 raft_consensus.cc:348] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.357844 6751 raft_consensus.cc:374] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:33.357894 6753 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: No bootstrap required, opened a new log
I20260810 14:38:33.358008 6751 raft_consensus.cc:729] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428884ca65c047929d3234cd7c596309, State: Initialized, Role: FOLLOWER
I20260810 14:38:33.358098 6753 ts_tablet_manager.cc:1397] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:38:33.358227 6754 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: No bootstrap required, opened a new log
I20260810 14:38:33.358417 6754 ts_tablet_manager.cc:1397] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:38:33.358832 6751 consensus_queue.cc:260] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.360181 6751 ts_tablet_manager.cc:1428] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:38:33.360646 6587 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
I20260810 14:38:33.362241 6753 raft_consensus.cc:348] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.362512 6753 raft_consensus.cc:374] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:33.362566 6753 raft_consensus.cc:729] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e1c648988949fc9f1250f6492c01e4, State: Initialized, Role: FOLLOWER
I20260810 14:38:33.363042 6753 consensus_queue.cc:260] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.363933 6734 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
I20260810 14:38:33.364231 6753 ts_tablet_manager.cc:1428] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 14:38:33.364054 6754 raft_consensus.cc:348] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.364305 6754 raft_consensus.cc:374] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:33.364357 6754 raft_consensus.cc:729] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eada159a0c404de5b69162d58221c6a0, State: Initialized, Role: FOLLOWER
I20260810 14:38:33.365020 6754 consensus_queue.cc:260] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.366001 6754 ts_tablet_manager.cc:1428] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:38:33.366104 6440 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
I20260810 14:38:33.375938 6757 raft_consensus.cc:482] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:33.376111 6757 raft_consensus.cc:504] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.377185 6757 leader_election.cc:288] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
I20260810 14:38:33.382025 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.377451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48119 (local address 127.21.43.65:39755)
0810 14:38:33.377578 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.377582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.377693 (+ 111us) server_negotiation.cc:407] Connection header received
0810 14:38:33.377854 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.377857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.377900 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.377966 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:33.378302 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.379080 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.380680 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.381333 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.381439 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.381497 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.381804 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.381877 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:38:33.382014 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.377391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.21.43.66:48119)
0810 14:38:33.377596 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.377612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.377744 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.378013 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.378016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.378042 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:33.378183 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.378193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.379211 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.379219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.380518 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.380529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.381449 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.381453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.381634 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.381692 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.381782 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.381841 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":112}
I20260810 14:38:33.382025 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.377503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:50965 (local address 127.21.43.67:36923)
0810 14:38:33.377708 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.377713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.378113 (+ 400us) server_negotiation.cc:407] Connection header received
0810 14:38:33.378271 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.378274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.378323 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.378398 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:33.378633 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.379329 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.380680 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.381308 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.381439 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.381497 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.381800 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.381873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 14:38:33.382157 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.377410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36923 (local address 127.21.43.66:50965)
0810 14:38:33.378017 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.378030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:33.378133 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:33.378421 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:33.378423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:33.378439 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:33.378526 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.378530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.379424 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.379427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.380562 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:33.380573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.381449 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.381453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:33.381620 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.381680 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 14:38:33.381767 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:38:33.381841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":195,"thread_start_us":132,"threads_started":1}
I20260810 14:38:33.382833 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9532c98c47fa4119b2d7d833664024c6" candidate_uuid: "428884ca65c047929d3234cd7c596309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e1c648988949fc9f1250f6492c01e4" is_pre_election: true
I20260810 14:38:33.382863 6385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9532c98c47fa4119b2d7d833664024c6" candidate_uuid: "428884ca65c047929d3234cd7c596309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eada159a0c404de5b69162d58221c6a0" is_pre_election: true
I20260810 14:38:33.383204 6385 raft_consensus.cc:2454] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428884ca65c047929d3234cd7c596309 in term 0.
I20260810 14:38:33.383216 6683 raft_consensus.cc:2454] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428884ca65c047929d3234cd7c596309 in term 0.
I20260810 14:38:33.383677 6462 leader_election.cc:302] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4; no voters:
I20260810 14:38:33.383991 6757 raft_consensus.cc:2781] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:33.384079 6757 raft_consensus.cc:482] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:33.384124 6757 raft_consensus.cc:3037] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:33.385350 6757 raft_consensus.cc:504] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.385696 6757 leader_election.cc:288] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [CANDIDATE]: Term 1 election: Requested vote from peers eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
I20260810 14:38:33.386030 6385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9532c98c47fa4119b2d7d833664024c6" candidate_uuid: "428884ca65c047929d3234cd7c596309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eada159a0c404de5b69162d58221c6a0"
I20260810 14:38:33.386080 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9532c98c47fa4119b2d7d833664024c6" candidate_uuid: "428884ca65c047929d3234cd7c596309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e1c648988949fc9f1250f6492c01e4"
I20260810 14:38:33.386160 6385 raft_consensus.cc:3037] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:33.386265 6683 raft_consensus.cc:3037] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:33.387432 6683 raft_consensus.cc:2454] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428884ca65c047929d3234cd7c596309 in term 1.
I20260810 14:38:33.387435 6385 raft_consensus.cc:2454] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428884ca65c047929d3234cd7c596309 in term 1.
I20260810 14:38:33.387775 6462 leader_election.cc:302] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4; no voters:
I20260810 14:38:33.388000 6757 raft_consensus.cc:2781] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:33.388269 6757 raft_consensus.cc:686] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 1 LEADER]: Becoming Leader. State: Replica: 428884ca65c047929d3234cd7c596309, State: Running, Role: LEADER
I20260810 14:38:33.388690 6757 consensus_queue.cc:237] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:33.392074 6221 catalog_manager.cc:5495] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428884ca65c047929d3234cd7c596309 (127.21.43.66). New cstate: current_term: 1 leader_uuid: "428884ca65c047929d3234cd7c596309" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260810 14:38:33.409350 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.405572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.21.43.66:45861)
0810 14:38:33.405716 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:33.405721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:33.405737 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:38:33.405857 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:33.405860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:33.405927 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:33.406041 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:33.406318 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.406841 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.407804 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:33.408026 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:33.408136 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:33.408467 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:33.408915 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:33.409052 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:33.409113 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 14:38:33.409173 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:38:33.415607 6683 raft_consensus.cc:1261] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Refusing update from remote peer 428884ca65c047929d3234cd7c596309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:33.415607 6385 raft_consensus.cc:1261] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Refusing update from remote peer 428884ca65c047929d3234cd7c596309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:33.416352 6761 consensus_queue.cc:1035] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:33.416574 6757 consensus_queue.cc:1035] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:33.424983 6769 mvcc.cc:205] Tried to move back new op lower bound from 7316982634142248960 to 7316982634042957824. Current Snapshot: MvccSnapshot[applied={T|T < 7316982634142248960}]
I20260810 14:38:33.425730 6767 mvcc.cc:205] Tried to move back new op lower bound from 7316982634142248960 to 7316982634042957824. Current Snapshot: MvccSnapshot[applied={T|T < 7316982634142248960}]
I20260810 14:38:33.435825 6218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:33.435999 6218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:33.438710 6218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Sending DeleteTablet for 3 replicas of tablet 9532c98c47fa4119b2d7d833664024c6
I20260810 14:38:33.439612 6511 tablet_service.cc:1514] Processing DeleteTablet for tablet 9532c98c47fa4119b2d7d833664024c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:33 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:38274
I20260810 14:38:33.439705 6360 tablet_service.cc:1514] Processing DeleteTablet for tablet 9532c98c47fa4119b2d7d833664024c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:33 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:40392
I20260810 14:38:33.440219 6778 tablet_replica.cc:330] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: stopping tablet replica
I20260810 14:38:33.440287 6779 tablet_replica.cc:330] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: stopping tablet replica
I20260810 14:38:33.440433 6662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9532c98c47fa4119b2d7d833664024c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:33 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:59328
I20260810 14:38:33.440649 6779 raft_consensus.cc:2229] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:33.440722 6778 raft_consensus.cc:2229] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:38:33.440968 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6292
I20260810 14:38:33.441033 6779 raft_consensus.cc:2258] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:33.444841 6778 raft_consensus.cc:2258] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:33.446920 6778 ts_tablet_manager.cc:1905] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:33.447404 6780 tablet_replica.cc:330] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: stopping tablet replica
I20260810 14:38:33.447825 6780 raft_consensus.cc:2229] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:33.448220 6780 raft_consensus.cc:2258] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:33.449766 6780 ts_tablet_manager.cc:1905] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:33.450659 6778 ts_tablet_manager.cc:1918] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:33.450773 6778 log.cc:1196] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/9532c98c47fa4119b2d7d833664024c6
I20260810 14:38:33.451128 6778 ts_tablet_manager.cc:1939] T 9532c98c47fa4119b2d7d833664024c6 P 428884ca65c047929d3234cd7c596309: Deleting consensus metadata
I20260810 14:38:33.452617 6206 catalog_manager.cc:4841] TS 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): tablet 9532c98c47fa4119b2d7d833664024c6 (table test-table [id=b83f8014b094417f83b2892d931f11b6]) successfully deleted
I20260810 14:38:33.452693 6780 ts_tablet_manager.cc:1918] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:33.452915 6780 log.cc:1196] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/9532c98c47fa4119b2d7d833664024c6
I20260810 14:38:33.453258 6780 ts_tablet_manager.cc:1939] T 9532c98c47fa4119b2d7d833664024c6 P 65e1c648988949fc9f1250f6492c01e4: Deleting consensus metadata
W20260810 14:38:33.453585 6203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:39755 (error 108)
I20260810 14:38:33.454443 6204 catalog_manager.cc:4841] TS 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): tablet 9532c98c47fa4119b2d7d833664024c6 (table test-table [id=b83f8014b094417f83b2892d931f11b6]) successfully deleted
I20260810 14:38:33.454712 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:33.454365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.0.0.1:40398)
0810 14:38:33.454526 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:33.454611 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:38:33.454869 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6445
W20260810 14:38:33.454931 6203 catalog_manager.cc:4584] TS eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9532c98c47fa4119b2d7d833664024c6: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39755: connect: Connection refused (error 111)
I20260810 14:38:33.464309 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6592
I20260810 14:38:33.472579 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6182
I20260810 14:38:53.482452 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--webserver_interface=127.21.43.126
--webserver_port=37605
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:38:53.604916 6782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:53.605291 6782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:53.605378 6782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:53.609648 6782 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:38:53.609748 6782 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:38:53.609778 6782 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:38:53.609803 6782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:38:53.609845 6782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:53.609892 6782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:53.609951 6782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:38:53.610044 6782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:38:53.616014 6782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:53.617245 6782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:53.618369 6782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:38:53.632879 6782 server_base.cc:1034] running on GCE node
W20260810 14:38:53.632828 6791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:53.632849 6790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:53.632828 6795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:53.633792 6782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:53.635074 6782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:53.636270 6782 hybrid_clock.cc:648] HybridClock initialized: now 1786372733636151 us; error 136 us; skew 500 ppm
Aug 10 14:38:53 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:53.644240 6782 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:38:53.645341 6782 webserver.cc:460] Webserver started at http://127.21.43.126:37605/ using document root <none> and password file <none>
I20260810 14:38:53.646128 6782 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:53.646258 6782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:53.650687 6782 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:38:53.653412 6802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:53.654819 6782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:53.655026 6782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "81db456f5d1b408db740a2444681e316"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "e4266b99d9b276926bf58ea8c14edb6d"
server_key_iv: "feab6b1b9aac7e1290d823c1f597ea7c"
server_key_version: "encryptionkey@0"
I20260810 14:38:53.655797 6782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:53.688099 6782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:53.692013 6782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:53.692281 6782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:53.700297 6782 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38751
I20260810 14:38:53.700631 6858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38751 every 8 connection(s)
I20260810 14:38:53.701601 6782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:38:53.702099 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6782
I20260810 14:38:53.702582 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:39755
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=40757
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:38:53.707248 6859 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:38:53.709194 6859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap starting.
I20260810 14:38:53.727269 6859 log.cc:824] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:53.740091 6859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:53.740543 6859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap complete.
I20260810 14:38:53.744926 6859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:53.745395 6859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db456f5d1b408db740a2444681e316, State: Initialized, Role: FOLLOWER
I20260810 14:38:53.745967 6859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:53.746121 6859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:38:53.746230 6859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:38:53.746357 6859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:38:53.748540 6859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:53.748908 6859 leader_election.cc:302] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db456f5d1b408db740a2444681e316; no voters:
I20260810 14:38:53.749274 6859 leader_election.cc:288] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:38:53.749488 6866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:38:53.749755 6866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 LEADER]: Becoming Leader. State: Replica: 81db456f5d1b408db740a2444681e316, State: Running, Role: LEADER
I20260810 14:38:53.750079 6866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:38:53.750710 6859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:38:53.751092 6866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db456f5d1b408db740a2444681e316. Latest consensus state: current_term: 2 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:38:53.751269 6866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:53.751933 6867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:38:53.752038 6867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:38:53.752725 6870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:38:53.758209 6870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:38:53.758843 6870 catalog_manager.cc:1259] Loaded cluster ID: cad90e6661374b96a44c403eaf7fa650
I20260810 14:38:53.758909 6870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:38:53.761368 6870 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:38:53.762585 6870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Loaded TSK: 0
I20260810 14:38:53.778443 6870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Generated new TSK 1
I20260810 14:38:53.778738 6870 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:38:53.837049 6862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:53.837347 6862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:53.837390 6862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:53.841758 6862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:53.841835 6862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:53.841928 6862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:38:53.847870 6862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=40757
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:53.848986 6862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:53.850049 6862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:53.860560 6895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:53.860612 6894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:53.860612 6897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:53.861423 6862 server_base.cc:1034] running on GCE node
I20260810 14:38:53.861886 6862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:53.862617 6862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:53.863806 6862 hybrid_clock.cc:648] HybridClock initialized: now 1786372733863776 us; error 45 us; skew 500 ppm
Aug 10 14:38:53 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:53.866808 6862 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:38:53.867904 6862 webserver.cc:460] Webserver started at http://127.21.43.65:40757/ using document root <none> and password file <none>
I20260810 14:38:53.868500 6862 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:53.868623 6862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:53.872970 6862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:53.875382 6904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:53.876785 6862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:53.876915 6862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "eada159a0c404de5b69162d58221c6a0"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "5c24bcdcd48b8d44ec24b47768a4336d"
server_key_iv: "a84f12520d2482a319409ff2042bd561"
server_key_version: "encryptionkey@0"
I20260810 14:38:53.877486 6862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:53.889333 6862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:53.892916 6862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:53.893133 6862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:53.893810 6862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:53.895026 6912 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:38:53.899257 6862 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:38:53.899326 6862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 14:38:53.899400 6862 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:38:53.901167 6862 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:38:53.901232 6862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 14:38:53.901510 6912 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Bootstrap starting.
I20260810 14:38:53.912918 6862 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39755
I20260810 14:38:53.912935 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39755 every 8 connection(s)
I20260810 14:38:53.914264 6862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:38:53.922847 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 6862
I20260810 14:38:53.923099 6912 log.cc:824] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:53.923435 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:45861
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=44509
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:53 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:53.932061 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:53.916525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:58067 (local address 127.21.43.126:38751)
0810 14:38:53.916911 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:53.916928 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:53.917366 (+ 438us) server_negotiation.cc:407] Connection header received
0810 14:38:53.918479 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:53.918490 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:53.918785 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:53.919115 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:53.920299 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.921292 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.922483 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.923281 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.923469 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:53.926644 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:53.926651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:53.926671 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:53.926711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:53.928735 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:53.929290 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:53.929296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:53.929301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:53.929432 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:53.929808 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:53.929812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:53.929814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:53.930173 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:53.930335 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:53.930870 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 14:38:53.931134 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":146,"threads_started":1}
I20260810 14:38:53.932073 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:53.916538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:58067)
0810 14:38:53.917176 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:53.917227 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:53.918056 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:53.919196 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:53.919212 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:53.919651 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:53.920100 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:53.920113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.921465 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.921468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:53.922348 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:53.922358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.923423 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.923427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:53.923828 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:53.924504 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:53.924530 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:53.926427 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:53.928873 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:53.928879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:53.928892 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:53.929163 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:53.929539 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:53.929543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:53.929545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:53.929696 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:53.930306 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:53.930312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:53.930560 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 14:38:53.930756 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 14:38:53.931108 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":167,"threads_started":1}
I20260810 14:38:53.932730 6912 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:38:53.933250 6912 tablet_bootstrap.cc:492] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Bootstrap complete.
I20260810 14:38:53.934662 7035 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:53.934880 6912 ts_tablet_manager.cc:1397] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.005s
I20260810 14:38:53.934911 7035 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:53.935524 7035 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:53.938100 6820 ts_manager.cc:194] Registered new tserver with Master: eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755)
I20260810 14:38:53.938984 6912 raft_consensus.cc:348] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:53.939561 6912 raft_consensus.cc:729] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eada159a0c404de5b69162d58221c6a0, State: Initialized, Role: FOLLOWER
I20260810 14:38:53.940090 6912 consensus_queue.cc:260] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } }
I20260810 14:38:53.941172 6912 ts_tablet_manager.cc:1428] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20260810 14:38:53.942451 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:58067
I20260810 14:38:53.945247 7035 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
Aug 10 14:38:53 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.65@KRBTEST.COM
I20260810 14:38:53.953474 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:53.941472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.0.0.1:38274)
0810 14:38:53.941897 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:53.941949 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:53.942112 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:53.943266 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:53.943271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:53.943772 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:53.943906 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:53.943914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.945009 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.945014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:53.945825 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:53.945834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.946640 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.946644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:53.946766 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:53.947481 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:53.947494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:53.949092 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:53.951441 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:53.951450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:53.951455 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:53.951737 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:53.952091 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:53.952094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:53.952096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:53.952204 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:53.952739 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:53.952746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:53.952902 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 14:38:53.953071 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 14:38:53.953291 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":129,"threads_started":1}
I20260810 14:38:53.953490 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:53.941630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.21.43.65:39755)
0810 14:38:53.942781 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:53.942793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:53.942817 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:38:53.942889 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:53.942895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:53.943089 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:53.943259 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:53.944057 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.944885 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.945955 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:53.946505 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:53.946618 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:53.949261 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:53.949272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:53.949287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:53.949314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:53.951294 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:53.951853 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:53.951861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:53.951863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:53.951964 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:53.952325 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:53.952328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:53.952329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:53.952608 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:53.952726 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:53.953210 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 14:38:53.953362 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":98,"threads_started":1}
I20260810 14:38:53.955212 6962 tablet_service.cc:1514] Processing DeleteTablet for tablet 9532c98c47fa4119b2d7d833664024c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:33 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:38274
I20260810 14:38:53.956269 7046 tablet_replica.cc:330] stopping tablet replica
I20260810 14:38:53.956605 7046 raft_consensus.cc:2229] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:53.956887 7046 raft_consensus.cc:2258] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:53.958010 7046 ts_tablet_manager.cc:1905] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:53.962101 7046 ts_tablet_manager.cc:1918] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:53.962190 7046 log.cc:1196] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal/wals/9532c98c47fa4119b2d7d833664024c6
I20260810 14:38:53.962469 7046 ts_tablet_manager.cc:1939] T 9532c98c47fa4119b2d7d833664024c6 P eada159a0c404de5b69162d58221c6a0: Deleting consensus metadata
I20260810 14:38:53.964684 6803 catalog_manager.cc:4841] TS eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755): tablet 9532c98c47fa4119b2d7d833664024c6 (table test-table [id=b83f8014b094417f83b2892d931f11b6]) successfully deleted
W20260810 14:38:54.051096 7040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:54.051505 7040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:54.051554 7040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:54.055933 7040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:54.056010 7040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:54.056123 7040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:38:54.061820 7040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:45861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=44509
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:54.062996 7040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:54.064073 7040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:38:54.073982 7054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:54.073973 7052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:54.073973 7051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:54.074378 7040 server_base.cc:1034] running on GCE node
I20260810 14:38:54.074874 7040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:54.075651 7040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:54.076880 7040 hybrid_clock.cc:648] HybridClock initialized: now 1786372734076844 us; error 49 us; skew 500 ppm
Aug 10 14:38:54 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372734, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:54.079891 7040 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:38:54.080834 7040 webserver.cc:460] Webserver started at http://127.21.43.66:44509/ using document root <none> and password file <none>
I20260810 14:38:54.081501 7040 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:54.081627 7040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:54.085769 7040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:54.087992 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.089303 7040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:38:54.089447 7040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "428884ca65c047929d3234cd7c596309"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "d6311add3a2a4a67081c369e67e56fcc"
server_key_iv: "bb14e5b0013f51ed8b01dd2865ad4b79"
server_key_version: "encryptionkey@0"
I20260810 14:38:54.090004 7040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:54.104561 7040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:54.107594 7040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:54.107816 7040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:54.108440 7040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:54.109295 7040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:54.109368 7040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.109443 7040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:54.109485 7040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.119359 7040 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45861
I20260810 14:38:54.119397 7186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45861 every 8 connection(s)
I20260810 14:38:54.120440 7040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:38:54.123371 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7040
I20260810 14:38:54.123871 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:36923
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=33325
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:38:54 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372734, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:54.136350 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.122364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:33957 (local address 127.21.43.126:38751)
0810 14:38:54.122549 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.122553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.123450 (+ 897us) server_negotiation.cc:407] Connection header received
0810 14:38:54.124927 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.124931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.124996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.125079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:54.126635 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.127231 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.129127 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.129414 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.129510 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.132541 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:54.132545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:54.132547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:54.132573 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:54.134126 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.134694 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.134697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.134698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.134777 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.135133 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.135155 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.135157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.135369 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:54.135480 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.136010 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.136170 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 14:38:54.137207 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.122756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:33957)
0810 14:38:54.123291 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.123335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.124706 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.125254 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.125261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.125722 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:54.126452 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.126474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.127373 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.127379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.128925 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.128940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.129600 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.129607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.129853 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.130544 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:54.130573 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:54.132349 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:54.134256 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:54.134262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:54.134278 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:54.134568 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:54.134872 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:54.134876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:54.134879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:54.135031 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:54.135480 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:54.135488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:54.135755 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.135955 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.136319 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260810 14:38:54.138548 7187 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:54.138813 7187 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:54.139457 7187 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:54.140632 6820 ts_manager.cc:194] Registered new tserver with Master: 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861)
I20260810 14:38:54.141414 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:33957
W20260810 14:38:54.243618 7190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:38:54.243901 7190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:38:54.243944 7190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:38:54.247985 7190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:38:54.248045 7190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:38:54.248104 7190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:38:54.253549 7190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:36923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=33325
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:38:54.254611 7190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:38:54.255646 7190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:38:54.266016 7190 server_base.cc:1034] running on GCE node
W20260810 14:38:54.266232 7195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:54.266232 7198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:38:54.266294 7196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:38:54.266777 7190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:38:54.267547 7190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:38:54.268724 7190 hybrid_clock.cc:648] HybridClock initialized: now 1786372734268693 us; error 47 us; skew 500 ppm
Aug 10 14:38:54 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372734, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:38:54.271842 7190 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:38:54.272802 7190 webserver.cc:460] Webserver started at http://127.21.43.67:33325/ using document root <none> and password file <none>
I20260810 14:38:54.273406 7190 fs_manager.cc:359] Metadata directory not provided
I20260810 14:38:54.273489 7190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:38:54.277565 7190 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:38:54.279837 7207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.281376 7190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:38:54.281529 7190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "65e1c648988949fc9f1250f6492c01e4"
format_stamp: "Formatted at 2026-08-10 14:38:33 on dist-test-slave-chsk"
server_key: "0d905a8689e1c26c868d403fe778be1e"
server_key_iv: "e7e9415adb87600acd4faee8ada171ae"
server_key_version: "encryptionkey@0"
I20260810 14:38:54.282156 7190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:38:54.315891 7190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:38:54.319126 7190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:38:54.319391 7190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:38:54.320058 7190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:38:54.320959 7190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:38:54.321046 7190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.321120 7190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:38:54.321170 7190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:38:54.331497 7190 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:36923
I20260810 14:38:54.331523 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:36923 every 8 connection(s)
I20260810 14:38:54.332598 7190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:38:54.332851 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7190
I20260810 14:38:54.342851 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.333663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.21.43.126:38751)
0810 14:38:54.333793 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.333796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.334158 (+ 362us) server_negotiation.cc:407] Connection header received
0810 14:38:54.334306 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.334310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.334370 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.334476 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:54.335426 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.336213 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.337503 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.337824 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.337896 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.339530 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:54.339534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:54.339537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:54.339563 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:54.341017 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.341491 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.341495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.341497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.341590 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.341923 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.341926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.341927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.342108 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:54.342291 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.342507 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.342653 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 14:38:54 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372734, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:38:54.347944 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.334679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:35371 (local address 127.21.43.126:38751)
0810 14:38:54.334955 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.334959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.335852 (+ 893us) server_negotiation.cc:407] Connection header received
0810 14:38:54.337505 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.337508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.337554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.337636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:54.339264 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.339962 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.341328 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.341642 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.341738 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.344456 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:54.344460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:54.344462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:54.344489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:54.345991 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.346528 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.346531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.346532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.346611 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.346918 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.346920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.346921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.347086 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:54.347265 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.347718 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.347818 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260810 14:38:54.348820 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.335204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:35371)
0810 14:38:54.335715 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.335757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.337297 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.337801 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.337808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.338356 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:38:54.339058 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.339098 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.340121 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.340123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.341181 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.341193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.341797 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.341801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.342020 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.342659 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:38:54.342679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:38:54.344298 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:38:54.346118 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:54.346123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:54.346135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:54.346421 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:54.346696 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:38:54.346698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:38:54.346699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:38:54.346830 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:38:54.347259 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:38:54.347265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:38:54.347500 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.347653 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.347912 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":151,"threads_started":1}
I20260810 14:38:54.349967 7332 heartbeater.cc:347] Connected to a master server at 127.21.43.126:38751
I20260810 14:38:54.350219 7332 heartbeater.cc:464] Registering TS with master...
I20260810 14:38:54.350749 7332 heartbeater.cc:511] Master 127.21.43.126:38751 requested a full tablet report, sending...
I20260810 14:38:54.351846 6820 ts_manager.cc:194] Registered new tserver with Master: 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
I20260810 14:38:54.352478 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:35371
I20260810 14:38:54.356612 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:38:54.360597 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.357248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.21.43.126:38751)
0810 14:38:54.357394 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.357399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.357562 (+ 163us) server_negotiation.cc:407] Connection header received
0810 14:38:54.357695 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.357697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.357792 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.357875 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:54.358081 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.358584 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.359525 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.359745 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.359826 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.360097 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:54.360240 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 14:38:54.360460 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56486: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 14:38:54.360675 7335 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56486: token expired
W20260810 14:38:54.360991 6738 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.43.126:38751)
I20260810 14:38:54.368456 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.361367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.21.43.126:38751)
0810 14:38:54.361481 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.361488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.361521 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:38:54.361651 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.361654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.361699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.361777 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:38:54.362420 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.362895 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.363770 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.363982 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.364087 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.365437 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:38:54.365441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:38:54.365443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:38:54.365468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:38:54.366769 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.367221 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.367224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.367226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.367312 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:38:54.367598 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:38:54.367602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:38:54.367603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:38:54.367782 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:38:54.367925 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.368165 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.368289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:38:54.370924 6820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:38:54.374255 6820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:38:54.382870 6962 tablet_service.cc:1467] Processing CreateTablet for tablet d20a084dd043491896de7bd553947c10 (DEFAULT_TABLE table=test-table [id=a51bbe4f01b74c9c9305155cadef726a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:54.383652 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20a084dd043491896de7bd553947c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:54.386787 6912 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Bootstrap starting.
I20260810 14:38:54.387323 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.381234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45861 (local address 127.0.0.1:54234)
0810 14:38:54.381411 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.381428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.381506 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.382717 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.382721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.382742 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:54.382867 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.382874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.384481 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.384484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.385722 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.385734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.386820 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.386824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.386920 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.386972 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.387030 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.387131 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 14:38:54.387705 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.381754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36923 (local address 127.0.0.1:36880)
0810 14:38:54.382028 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.382043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.382221 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.382955 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.382959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.382980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:54.383091 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.383098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.384739 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.384742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.385623 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.385632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.387025 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.387029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.387128 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.387188 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.387263 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.387326 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260810 14:38:54.387799 6912 tablet_bootstrap.cc:654] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:54.387917 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.381429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.21.43.66:45861)
0810 14:38:54.382010 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.382024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.382044 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:38:54.382149 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.382157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.382488 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.382746 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:54.383016 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.384346 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.385844 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.386671 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.387434 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.387600 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.387651 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.387749 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":255,"threads_started":1}
I20260810 14:38:54.388020 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.381935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.21.43.67:36923)
0810 14:38:54.382448 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.382461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.382483 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:38:54.382559 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.382564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.382751 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.382903 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:54.383408 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.384622 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.385746 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.386705 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.386807 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.386959 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.387323 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.387637 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":203,"threads_started":1}
I20260810 14:38:54.389437 6912 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: No bootstrap required, opened a new log
I20260810 14:38:54.389534 6912 ts_tablet_manager.cc:1397] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:38:54.390069 6912 raft_consensus.cc:348] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.390194 6912 raft_consensus.cc:374] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:54.390232 6912 raft_consensus.cc:729] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eada159a0c404de5b69162d58221c6a0, State: Initialized, Role: FOLLOWER
I20260810 14:38:54.390365 6912 consensus_queue.cc:260] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.391273 6912 ts_tablet_manager.cc:1428] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:38:54.391422 7258 tablet_service.cc:1467] Processing CreateTablet for tablet d20a084dd043491896de7bd553947c10 (DEFAULT_TABLE table=test-table [id=a51bbe4f01b74c9c9305155cadef726a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:54.391402 7113 tablet_service.cc:1467] Processing CreateTablet for tablet d20a084dd043491896de7bd553947c10 (DEFAULT_TABLE table=test-table [id=a51bbe4f01b74c9c9305155cadef726a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:38:54.392398 7113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20a084dd043491896de7bd553947c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:54.392406 7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d20a084dd043491896de7bd553947c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:38:54.399670 7347 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Bootstrap starting.
I20260810 14:38:54.400369 7346 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Bootstrap starting.
I20260810 14:38:54.402232 7347 tablet_bootstrap.cc:654] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:54.402652 7346 tablet_bootstrap.cc:654] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Neither blocks nor log segments found. Creating new log.
I20260810 14:38:54.403439 7346 log.cc:824] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:54.403440 7347 log.cc:824] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:38:54.405334 7346 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: No bootstrap required, opened a new log
I20260810 14:38:54.405387 7347 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: No bootstrap required, opened a new log
I20260810 14:38:54.405560 7346 ts_tablet_manager.cc:1397] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:38:54.405576 7347 ts_tablet_manager.cc:1397] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:38:54.408562 7347 raft_consensus.cc:348] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.408561 7346 raft_consensus.cc:348] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.408802 7347 raft_consensus.cc:374] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:54.408836 7346 raft_consensus.cc:374] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:38:54.408838 7347 raft_consensus.cc:729] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e1c648988949fc9f1250f6492c01e4, State: Initialized, Role: FOLLOWER
I20260810 14:38:54.408936 7346 raft_consensus.cc:729] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428884ca65c047929d3234cd7c596309, State: Initialized, Role: FOLLOWER
I20260810 14:38:54.409353 7347 consensus_queue.cc:260] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.409363 7346 consensus_queue.cc:260] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.410096 7187 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
I20260810 14:38:54.410311 7332 heartbeater.cc:503] Master 127.21.43.126:38751 was elected leader, sending a full tablet report...
I20260810 14:38:54.410401 7346 ts_tablet_manager.cc:1428] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 14:38:54.410403 7347 ts_tablet_manager.cc:1428] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 14:38:54.415529 7036 tablet.cc:2369] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:38:54.492966 7352 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:38:54.493247 7352 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.494760 7352 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755)
I20260810 14:38:54.500109 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.495112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:50893 (local address 127.21.43.66:45861)
0810 14:38:54.495269 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.495275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.495387 (+ 112us) server_negotiation.cc:407] Connection header received
0810 14:38:54.495506 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.495509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.495603 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.495694 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:54.496104 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.496906 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.498802 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.499427 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.499570 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.499646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.499898 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.499977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:38:54.500110 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.495193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:48559 (local address 127.21.43.65:39755)
0810 14:38:54.495486 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.495490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.495506 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:38:54.495653 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.495656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.495727 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.495851 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:38:54.496139 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.496906 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.498802 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.499545 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.499657 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.499817 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.499903 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.499978 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260810 14:38:54.500109 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.495019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45861 (local address 127.21.43.67:50893)
0810 14:38:54.495269 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.495285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.495392 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.495764 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.495767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.495786 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:54.495940 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.495949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.497038 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.497042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.498640 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.498653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.499571 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.499573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.499710 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.499740 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.499821 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.499942 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":247}
I20260810 14:38:54.500118 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.495053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.21.43.67:48559)
0810 14:38:54.495440 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.495456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:38:54.495541 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:38:54.495879 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:38:54.495882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:38:54.495897 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:38:54.496014 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.496025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.497038 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.497042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.498640 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:38:54.498653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.499650 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.499653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:38:54.499742 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.499759 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 14:38:54.499821 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:38:54.499934 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":198,"thread_start_us":152,"threads_started":1}
I20260810 14:38:54.500941 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20a084dd043491896de7bd553947c10" candidate_uuid: "65e1c648988949fc9f1250f6492c01e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428884ca65c047929d3234cd7c596309" is_pre_election: true
I20260810 14:38:54.501012 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20a084dd043491896de7bd553947c10" candidate_uuid: "65e1c648988949fc9f1250f6492c01e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eada159a0c404de5b69162d58221c6a0" is_pre_election: true
I20260810 14:38:54.501304 6985 raft_consensus.cc:2454] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e1c648988949fc9f1250f6492c01e4 in term 0.
I20260810 14:38:54.501304 7136 raft_consensus.cc:2454] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e1c648988949fc9f1250f6492c01e4 in term 0.
I20260810 14:38:54.501914 7208 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4, eada159a0c404de5b69162d58221c6a0; no voters:
I20260810 14:38:54.502243 7352 raft_consensus.cc:2781] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:38:54.502333 7352 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:38:54.502379 7352 raft_consensus.cc:3037] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:54.503693 7352 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.504508 7352 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [CANDIDATE]: Term 1 election: Requested vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755)
I20260810 14:38:54.504920 6985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20a084dd043491896de7bd553947c10" candidate_uuid: "65e1c648988949fc9f1250f6492c01e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eada159a0c404de5b69162d58221c6a0"
I20260810 14:38:54.504969 7136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d20a084dd043491896de7bd553947c10" candidate_uuid: "65e1c648988949fc9f1250f6492c01e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "428884ca65c047929d3234cd7c596309"
I20260810 14:38:54.505097 7136 raft_consensus.cc:3037] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:54.505095 6985 raft_consensus.cc:3037] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:38:54.506148 6985 raft_consensus.cc:2454] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e1c648988949fc9f1250f6492c01e4 in term 1.
I20260810 14:38:54.506160 7136 raft_consensus.cc:2454] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e1c648988949fc9f1250f6492c01e4 in term 1.
I20260810 14:38:54.506474 7208 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e1c648988949fc9f1250f6492c01e4, eada159a0c404de5b69162d58221c6a0; no voters:
I20260810 14:38:54.506685 7352 raft_consensus.cc:2781] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:38:54.506930 7352 raft_consensus.cc:686] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 1 LEADER]: Becoming Leader. State: Replica: 65e1c648988949fc9f1250f6492c01e4, State: Running, Role: LEADER
I20260810 14:38:54.507350 7352 consensus_queue.cc:237] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:38:54.510735 6820 catalog_manager.cc:5495] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "65e1c648988949fc9f1250f6492c01e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260810 14:38:54.543454 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.539808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.21.43.67:36923)
0810 14:38:54.539961 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:38:54.539965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:38:54.539979 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:38:54.540066 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:38:54.540068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:38:54.540125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:38:54.540223 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:38:54.540528 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.541034 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.541913 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:38:54.542159 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:38:54.542245 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:38:54.542533 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:38:54.542951 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:38:54.543048 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:38:54.543255 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 14:38:54.543314 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:38:54.549871 6985 raft_consensus.cc:1261] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Refusing update from remote peer 65e1c648988949fc9f1250f6492c01e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:54.549875 7136 raft_consensus.cc:1261] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Refusing update from remote peer 65e1c648988949fc9f1250f6492c01e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:38:54.550691 7356 consensus_queue.cc:1035] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:54.550902 7352 consensus_queue.cc:1035] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:38:54.557734 7359 mvcc.cc:205] Tried to move back new op lower bound from 7316982720708337664 to 7316982720545140736. Current Snapshot: MvccSnapshot[applied={T|T < 7316982720708337664}]
I20260810 14:38:54.567612 6820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:54.567813 6820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:38:54.570248 6820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Sending DeleteTablet for 3 replicas of tablet d20a084dd043491896de7bd553947c10
I20260810 14:38:54.571048 7113 tablet_service.cc:1514] Processing DeleteTablet for tablet d20a084dd043491896de7bd553947c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:54 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:54234
I20260810 14:38:54.571237 6962 tablet_service.cc:1514] Processing DeleteTablet for tablet d20a084dd043491896de7bd553947c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:54 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:38274
I20260810 14:38:54.571702 7370 tablet_replica.cc:330] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: stopping tablet replica
I20260810 14:38:54.571722 7258 tablet_service.cc:1514] Processing DeleteTablet for tablet d20a084dd043491896de7bd553947c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:38:54 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:36880
I20260810 14:38:54.572085 7370 raft_consensus.cc:2229] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:38:54.572221 7371 tablet_replica.cc:330] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: stopping tablet replica
I20260810 14:38:54.572371 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6862
I20260810 14:38:54.572451 7370 raft_consensus.cc:2258] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:54.572661 7371 raft_consensus.cc:2229] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:38:54.574191 7371 raft_consensus.cc:2258] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:38:54.576113 7371 ts_tablet_manager.cc:1905] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:54.578516 7370 ts_tablet_manager.cc:1905] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:38:54.579349 7371 ts_tablet_manager.cc:1918] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:54.579541 7371 log.cc:1196] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/d20a084dd043491896de7bd553947c10
I20260810 14:38:54.579977 7371 ts_tablet_manager.cc:1939] T d20a084dd043491896de7bd553947c10 P 65e1c648988949fc9f1250f6492c01e4: Deleting consensus metadata
I20260810 14:38:54.581429 6804 catalog_manager.cc:4841] TS 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): tablet d20a084dd043491896de7bd553947c10 (table test-table [id=a51bbe4f01b74c9c9305155cadef726a]) successfully deleted
I20260810 14:38:54.582157 7370 ts_tablet_manager.cc:1918] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:38:54.582274 7370 log.cc:1196] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/d20a084dd043491896de7bd553947c10
I20260810 14:38:54.582808 7370 ts_tablet_manager.cc:1939] T d20a084dd043491896de7bd553947c10 P 428884ca65c047929d3234cd7c596309: Deleting consensus metadata
I20260810 14:38:54.584380 6806 catalog_manager.cc:4841] TS 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): tablet d20a084dd043491896de7bd553947c10 (table test-table [id=a51bbe4f01b74c9c9305155cadef726a]) successfully deleted
W20260810 14:38:54.585788 6803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:39755 (error 108)
I20260810 14:38:54.586498 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 7040
I20260810 14:38:54.587994 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:38:54.586670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:39755 (local address 127.0.0.1:38288)
0810 14:38:54.587797 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0810 14:38:54.587882 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":996}
W20260810 14:38:54.588303 6803 catalog_manager.cc:4584] TS eada159a0c404de5b69162d58221c6a0 (127.21.43.65:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d20a084dd043491896de7bd553947c10: Network error: Client connection negotiation failed: client connection to 127.21.43.65:39755: connect: Connection refused (error 111)
I20260810 14:38:54.595553 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 7190
I20260810 14:38:54.604049 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 6782
I20260810 14:39:14.614689 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--webserver_interface=127.21.43.126
--webserver_port=37605
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:38751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:14.750037 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:14.750337 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:14.750378 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:14.754453 7374 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:39:14.754516 7374 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:39:14.754534 7374 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:39:14.754547 7374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:14.754597 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:14.754638 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:14.754659 7374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:14.754691 7374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:14.760397 7374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:38751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:38751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=37605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:14.761622 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:14.762763 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:14.771718 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:14.771775 7374 server_base.cc:1034] running on GCE node
W20260810 14:39:14.771718 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:14.771910 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:14.772642 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:14.773837 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:14.775020 7374 hybrid_clock.cc:648] HybridClock initialized: now 1786372754775007 us; error 34 us; skew 500 ppm
Aug 10 14:39:14 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372754, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:14.783485 7374 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:39:14.784608 7374 webserver.cc:460] Webserver started at http://127.21.43.126:37605/ using document root <none> and password file <none>
I20260810 14:39:14.785205 7374 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:14.785303 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:14.789501 7374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:39:14.791968 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:14.793452 7374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:14.793628 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "81db456f5d1b408db740a2444681e316"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "e4266b99d9b276926bf58ea8c14edb6d"
server_key_iv: "feab6b1b9aac7e1290d823c1f597ea7c"
server_key_version: "encryptionkey@0"
I20260810 14:39:14.794260 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:14.816111 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:14.819427 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:14.819669 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:14.827673 7374 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:38751
I20260810 14:39:14.828055 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:38751 every 8 connection(s)
I20260810 14:39:14.828890 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:14.833878 7447 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:39:14.835367 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7374
I20260810 14:39:14.835762 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap starting.
I20260810 14:39:14.835898 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:39755
--local_ip_for_outbound_sockets=127.21.43.65
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=40757
--webserver_interface=127.21.43.65
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:39:14.853660 7447 log.cc:824] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:14.865147 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:39:14.865563 7447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Bootstrap complete.
I20260810 14:39:14.869671 7447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:39:14.870085 7447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db456f5d1b408db740a2444681e316, State: Initialized, Role: FOLLOWER
I20260810 14:39:14.870577 7447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:39:14.870733 7447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:14.870826 7447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:14.870954 7447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:39:14.872913 7447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:39:14.873307 7447 leader_election.cc:302] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db456f5d1b408db740a2444681e316; no voters:
I20260810 14:39:14.873728 7447 leader_election.cc:288] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:39:14.873817 7452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:39:14.874035 7452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [term 3 LEADER]: Becoming Leader. State: Replica: 81db456f5d1b408db740a2444681e316, State: Running, Role: LEADER
I20260810 14:39:14.874421 7452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } }
I20260810 14:39:14.875025 7447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:14.875433 7452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db456f5d1b408db740a2444681e316. Latest consensus state: current_term: 3 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:39:14.875648 7452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:14.875844 7453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "81db456f5d1b408db740a2444681e316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db456f5d1b408db740a2444681e316" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 38751 } } }
I20260810 14:39:14.875964 7453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:14.876233 7457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:14.881798 7457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:14.882588 7457 catalog_manager.cc:1259] Loaded cluster ID: cad90e6661374b96a44c403eaf7fa650
I20260810 14:39:14.882679 7457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:14.884585 7457 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:14.885206 7457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Loaded TSK: 0
I20260810 14:39:14.885301 7457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81db456f5d1b408db740a2444681e316: Loaded TSK: 1
F20260810 14:39:14.885962 7457 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90d2f459ad google::LogMessage::Fail() at ??:0
@ 0x7f90d2f4888a google::LogMessage::SendToLog() at ??:0
@ 0x7f90d2f454de google::LogMessage::Flush() at ??:0
@ 0x7f90d2f46a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f90d5c0918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f90d5c01f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f90dd8aa7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f90dd88499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f90dd885178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f90dd8c6df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f90dd8e582b std::function<>::operator()() at ??:0
@ 0x7f90dd884b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f90dd885f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f90dd882579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f90dd8c6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90d51973d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90d5197c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f90d51993f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f90d5188512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f90d67cc6db start_thread at ??:0
@ 0x7f90d1a4361f clone at ??:0
W20260810 14:39:14.985184 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:14.985529 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:14.985615 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:14.992395 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:14.992492 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:14.992592 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:15.000644 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=40757
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:15.002190 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:15.003721 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:15.015281 7499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.015383 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.015290 7498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:15.015951 7450 server_base.cc:1034] running on GCE node
I20260810 14:39:15.016412 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:15.017206 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:15.018429 7450 hybrid_clock.cc:648] HybridClock initialized: now 1786372755018403 us; error 32 us; skew 500 ppm
Aug 10 14:39:15 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:15.021944 7450 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:39:15.023073 7450 webserver.cc:460] Webserver started at http://127.21.43.65:40757/ using document root <none> and password file <none>
I20260810 14:39:15.023831 7450 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:15.023928 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:15.029095 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:39:15.031834 7511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.033344 7450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:15.033474 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "eada159a0c404de5b69162d58221c6a0"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "5c24bcdcd48b8d44ec24b47768a4336d"
server_key_iv: "a84f12520d2482a319409ff2042bd561"
server_key_version: "encryptionkey@0"
I20260810 14:39:15.034216 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:15.058554 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:15.062567 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:15.062795 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:15.063669 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:15.065101 7518 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 14:39:15.069437 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:39:15.069520 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 14:39:15.069599 7450 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:39:15.071893 7450 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:39:15.071959 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:15.072288 7518 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Bootstrap starting.
I20260810 14:39:15.083487 7450 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:39755
I20260810 14:39:15.083840 7636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:39755 every 8 connection(s)
I20260810 14:39:15.084733 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:15.092792 7518 log.cc:824] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:15.094785 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7450
I20260810 14:39:15.095335 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:45861
--local_ip_for_outbound_sockets=127.21.43.66
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=44509
--webserver_interface=127.21.43.66
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:39:15.101024 7518 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:39:15.101442 7518 tablet_bootstrap.cc:492] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Bootstrap complete.
I20260810 14:39:15.102664 7518 ts_tablet_manager.cc:1397] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.013s
I20260810 14:39:15.105239 7518 raft_consensus.cc:348] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:15.105684 7518 raft_consensus.cc:729] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eada159a0c404de5b69162d58221c6a0, State: Initialized, Role: FOLLOWER
I20260810 14:39:15.106194 7518 consensus_queue.cc:260] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:15.107240 7518 ts_tablet_manager.cc:1428] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260810 14:39:15.170130 7640 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:39:15.087167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:36213)
0810 14:39:15.087828 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.087888 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:15.089111 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:15.169314 (+ 80203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":156,"threads_started":1}
W20260810 14:39:15.170763 7638 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:38751 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:38751: Client connection negotiation failed: client connection to 127.21.43.126:38751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:39:15.171590 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.171319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:40087)
0810 14:39:15.171451 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.171532 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 14:39:15.172363 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.172156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:54785)
0810 14:39:15.172267 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.172311 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 14:39:15.172585 7638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:39:15.237282 7642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:15.237571 7642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:15.237664 7642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:15.242050 7642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:15.242125 7642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:15.242187 7642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:15.247730 7642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:45861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=44509
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:15.248878 7642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:15.249927 7642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:15.263409 7650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.263902 7653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.263957 7651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:15.264729 7642 server_base.cc:1034] running on GCE node
I20260810 14:39:15.265276 7642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:15.266255 7642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:15.267459 7642 hybrid_clock.cc:648] HybridClock initialized: now 1786372755267423 us; error 42 us; skew 500 ppm
Aug 10 14:39:15 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:15.271697 7642 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:39:15.272748 7642 webserver.cc:460] Webserver started at http://127.21.43.66:44509/ using document root <none> and password file <none>
I20260810 14:39:15.273413 7642 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:15.273504 7642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:15.278821 7642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:15.281917 7662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.283891 7642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260810 14:39:15.284044 7642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "428884ca65c047929d3234cd7c596309"
format_stamp: "Formatted at 2026-08-10 14:38:32 on dist-test-slave-chsk"
server_key: "d6311add3a2a4a67081c369e67e56fcc"
server_key_iv: "bb14e5b0013f51ed8b01dd2865ad4b79"
server_key_version: "encryptionkey@0"
I20260810 14:39:15.284911 7642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:15.312060 7642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:15.315723 7642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:15.315958 7642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:15.316627 7642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:15.317524 7642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:15.317575 7642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.317664 7642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:15.317698 7642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.327767 7642 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45861
I20260810 14:39:15.327813 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45861 every 8 connection(s)
I20260810 14:39:15.328845 7642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:15.333048 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.331102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:41757)
0810 14:39:15.331695 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.331812 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":124,"threads_started":1}
W20260810 14:39:15.333534 7790 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:38751 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:38751: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
I20260810 14:39:15.334194 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.333953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:49207)
0810 14:39:15.334077 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.334132 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:39:15.334796 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.334600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:37041)
0810 14:39:15.334704 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.334746 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:39:15.334985 7790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:39:15.338049 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7642
I20260810 14:39:15.338560 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:36923
--local_ip_for_outbound_sockets=127.21.43.67
--tserver_master_addrs=127.21.43.126:38751
--webserver_port=33325
--webserver_interface=127.21.43.67
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:15.459240 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:15.459556 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:15.459604 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:15.463740 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:15.463814 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:15.463879 7794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:15.469389 7794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:36923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=33325
--tserver_master_addrs=127.21.43.126:38751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:15.470513 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:15.471599 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:15.481912 7800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.481940 7799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:15.481962 7802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:15.482441 7794 server_base.cc:1034] running on GCE node
I20260810 14:39:15.482952 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:15.483726 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:15.484912 7794 hybrid_clock.cc:648] HybridClock initialized: now 1786372755484872 us; error 51 us; skew 500 ppm
Aug 10 14:39:15 dist-test-slave-chsk krb5kdc[6168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:15.487946 7794 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:39:15.488998 7794 webserver.cc:460] Webserver started at http://127.21.43.67:33325/ using document root <none> and password file <none>
I20260810 14:39:15.489709 7794 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:15.489780 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:15.494134 7794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:15.496590 7810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.498056 7794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:39:15.498180 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "65e1c648988949fc9f1250f6492c01e4"
format_stamp: "Formatted at 2026-08-10 14:38:33 on dist-test-slave-chsk"
server_key: "0d905a8689e1c26c868d403fe778be1e"
server_key_iv: "e7e9415adb87600acd4faee8ada171ae"
server_key_version: "encryptionkey@0"
I20260810 14:39:15.498821 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:15.526342 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:15.529778 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:15.530051 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:15.530748 7794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:15.531827 7794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:15.531879 7794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.531922 7794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:15.531988 7794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:15.542564 7794 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:36923
I20260810 14:39:15.542590 7936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:36923 every 8 connection(s)
I20260810 14:39:15.543735 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:15.547541 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 7794
I20260810 14:39:15.548259 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.546078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:55187)
0810 14:39:15.546597 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.546719 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":111,"threads_started":1}
W20260810 14:39:15.548914 7937 heartbeater.cc:644] Failed to heartbeat to 127.21.43.126:38751 (0 consecutive failures): Network error: Failed to ping master at 127.21.43.126:38751: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
I20260810 14:39:15.549532 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.549303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:46431)
0810 14:39:15.549424 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.549472 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:39:15.550226 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:15.550001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:47851)
0810 14:39:15.550108 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:15.550163 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:39:15.550395 7937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:39:16.173979 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.173434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:52003)
0810 14:39:16.173793 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:16.173869 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":99,"threads_started":1}
I20260810 14:39:16.336176 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.335657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:57739)
0810 14:39:16.335983 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:16.336048 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260810 14:39:16.551718 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.551116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:58835)
0810 14:39:16.551542 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:16.551606 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":73,"threads_started":1}
I20260810 14:39:16.762445 7946 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:16.762756 7946 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:16.764062 7946 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
Aug 10 14:39:16 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.66@KRBTEST.COM
Aug 10 14:39:16 dist-test-slave-chsk krb5kdc[6168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.67@KRBTEST.COM
I20260810 14:39:16.782639 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.764599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:42425 (local address 127.21.43.67:36923)
0810 14:39:16.765056 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:16.765086 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:16.765135 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:39:16.766077 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:16.766087 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:16.766565 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:16.766918 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:16.768922 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.770516 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.772567 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.773336 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.773532 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:16.777374 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:16.777383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:16.777402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:16.777471 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:16.780003 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780512 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:16.780518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:16.780523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:16.780669 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780982 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:16.780985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:16.780987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:16.781424 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:16.781582 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:16.782177 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 14:39:16.782445 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":94,"threads_started":1}
I20260810 14:39:16.782630 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.764338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36923 (local address 127.21.43.65:42425)
0810 14:39:16.764637 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:16.764654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:16.764788 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:16.767067 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:16.767075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:16.767829 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:16.768692 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:16.768717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.770728 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.770732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:16.772389 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:16.772402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.773513 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.773518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:16.773807 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:16.774700 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:16.774727 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:16.777096 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:16.780147 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:16.780156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:16.780373 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:16.780781 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:16.780786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:16.780875 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:16.781555 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:16.781568 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:16.781816 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 14:39:16.782035 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 14:39:16.782418 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":439,"spinlock_wait_cycles":36224,"thread_start_us":70,"threads_started":1}
I20260810 14:39:16.782676 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.764349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45861 (local address 127.21.43.65:53039)
0810 14:39:16.764730 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:16.764743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:16.764803 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:16.767067 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:16.767075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:16.767829 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:16.768692 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:16.768717 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.770728 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.770732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:16.772390 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:16.772402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.773513 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.773518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:16.773846 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:16.774700 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:16.774740 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:16.776913 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:16.779955 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:16.779962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:16.779977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:16.780283 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:16.780682 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:16.780689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:16.780855 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:16.781554 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:16.781561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:16.781816 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 14:39:16.782035 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 14:39:16.782507 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":469,"spinlock_wait_cycles":65664,"thread_start_us":82,"threads_started":1}
I20260810 14:39:16.783423 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:16.764671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:53039 (local address 127.21.43.66:45861)
0810 14:39:16.765072 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:16.765093 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:16.765135 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:39:16.766077 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:16.766087 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:16.766565 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:16.766918 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:16.768923 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.770515 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.772567 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:16.773336 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:16.773530 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:16.777202 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:16.777208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:16.777222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:16.777258 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:16.779806 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780435 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:16.780441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:16.780445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:16.780574 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:16.780971 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:16.780975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:16.780981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:16.781424 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:16.781590 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:16.782169 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 14:39:16.782483 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":84,"threads_started":1}
W20260810 14:39:16.785588 7513 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
W20260810 14:39:16.785836 7515 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
I20260810 14:39:16.785915 7515 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eada159a0c404de5b69162d58221c6a0; no voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4
I20260810 14:39:16.786278 7946 raft_consensus.cc:2726] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:39:17.175374 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:17.174962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:40817)
0810 14:39:17.175216 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:17.175283 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 14:39:17.337502 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:17.337057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:49253)
0810 14:39:17.337351 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:17.337409 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":80,"threads_started":1}
I20260810 14:39:17.553002 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:17.552557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:54275)
0810 14:39:17.552840 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:17.552898 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":109,"threads_started":1}
I20260810 14:39:18.176781 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:18.176305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:55831)
0810 14:39:18.176596 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:18.176675 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260810 14:39:18.338758 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:18.338314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:47511)
0810 14:39:18.338593 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:18.338650 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260810 14:39:18.554360 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:18.553851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:60491)
0810 14:39:18.554180 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:18.554254 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260810 14:39:18.936923 7958 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:18.937083 7958 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:18.937638 7958 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
W20260810 14:39:18.938539 7513 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
W20260810 14:39:18.938638 7515 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
I20260810 14:39:18.938732 7515 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eada159a0c404de5b69162d58221c6a0; no voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4
I20260810 14:39:18.938868 7958 raft_consensus.cc:2726] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:39:19.178334 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:19.177777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:37681)
0810 14:39:19.178162 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:19.178230 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":97,"threads_started":1}
I20260810 14:39:19.340435 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:19.339787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:57039)
0810 14:39:19.340182 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:19.340300 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":150,"threads_started":1}
I20260810 14:39:19.555708 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:19.555246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:36289)
0810 14:39:19.555543 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:19.555606 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260810 14:39:20.179981 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:20.179484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:60785)
0810 14:39:20.179817 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:20.179880 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260810 14:39:20.341710 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:20.341256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:39359)
0810 14:39:20.341546 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:20.341608 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260810 14:39:20.557092 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:20.556624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:38211)
0810 14:39:20.556922 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:20.556989 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260810 14:39:21.181519 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:21.181015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:57979)
0810 14:39:21.181332 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:21.181400 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20260810 14:39:21.342917 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:21.342491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:59395)
0810 14:39:21.342762 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:21.342816 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
I20260810 14:39:21.362308 7968 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:21.362455 7968 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:21.362988 7968 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
W20260810 14:39:21.363904 7515 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
W20260810 14:39:21.363997 7513 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
I20260810 14:39:21.364104 7513 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eada159a0c404de5b69162d58221c6a0; no voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4
I20260810 14:39:21.364255 7968 raft_consensus.cc:2726] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:39:21.558573 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:21.558073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:53541)
0810 14:39:21.558397 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:21.558468 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260810 14:39:22.182792 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:22.182353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:54303)
0810 14:39:22.182635 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:22.182694 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260810 14:39:22.344434 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:22.343880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:40713)
0810 14:39:22.344255 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:22.344329 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":116,"threads_started":1}
I20260810 14:39:22.559973 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:22.559467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:50849)
0810 14:39:22.559809 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:22.559866 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260810 14:39:23.184160 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:23.183654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:59353)
0810 14:39:23.183960 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:23.184054 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260810 14:39:23.345965 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:23.345454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:59271)
0810 14:39:23.345790 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:23.345859 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260810 14:39:23.561662 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:23.561083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:59895)
0810 14:39:23.561461 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:23.561522 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260810 14:39:24.185575 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:24.185067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:46771)
0810 14:39:24.185409 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:24.185471 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":89,"threads_started":1}
I20260810 14:39:24.347658 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:24.346957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:49013)
0810 14:39:24.347411 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:24.347505 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":123,"threads_started":1}
I20260810 14:39:24.562929 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:24.562491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:50825)
0810 14:39:24.562773 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:24.562830 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
I20260810 14:39:25.187238 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:25.186663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:49121)
0810 14:39:25.186992 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:25.187056 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260810 14:39:25.349262 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:25.348617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:57405)
0810 14:39:25.348988 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:25.349085 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260810 14:39:25.564383 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:25.563883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:54777)
0810 14:39:25.564206 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:25.564277 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20260810 14:39:25.996381 7984 raft_consensus.cc:482] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:25.996522 7984 raft_consensus.cc:504] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428884ca65c047929d3234cd7c596309" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45861 } } peers { permanent_uuid: "65e1c648988949fc9f1250f6492c01e4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36923 } } peers { permanent_uuid: "eada159a0c404de5b69162d58221c6a0" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 39755 } }
I20260810 14:39:25.996994 7984 leader_election.cc:288] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861), 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923)
W20260810 14:39:25.997888 7513 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65e1c648988949fc9f1250f6492c01e4 (127.21.43.67:36923): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
W20260810 14:39:25.998003 7515 leader_election.cc:341] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 428884ca65c047929d3234cd7c596309 (127.21.43.66:45861): Not found: Tablet not found: d20a084dd043491896de7bd553947c10
I20260810 14:39:25.998070 7515 leader_election.cc:302] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eada159a0c404de5b69162d58221c6a0; no voters: 428884ca65c047929d3234cd7c596309, 65e1c648988949fc9f1250f6492c01e4
I20260810 14:39:25.998205 7984 raft_consensus.cc:2726] T d20a084dd043491896de7bd553947c10 P eada159a0c404de5b69162d58221c6a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:39:26.188632 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:26.188161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:43313)
0810 14:39:26.188458 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:26.188528 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20260810 14:39:26.350677 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:26.350198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:59641)
0810 14:39:26.350506 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:26.350573 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260810 14:39:26.565915 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:26.565402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:32779)
0810 14:39:26.565734 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:26.565809 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260810 14:39:27.190039 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:27.189565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:37275)
0810 14:39:27.189849 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:27.189930 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260810 14:39:27.351888 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:27.351476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:41405)
0810 14:39:27.351734 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:27.351788 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260810 14:39:27.567526 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:27.566937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:52959)
0810 14:39:27.567358 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:27.567422 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":103,"threads_started":1}
I20260810 14:39:28.191390 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:28.190884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:60885)
0810 14:39:28.191227 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:28.191287 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260810 14:39:28.353240 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:28.352741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:39383)
0810 14:39:28.353055 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:28.353128 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260810 14:39:28.568733 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:28.568319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:36753)
0810 14:39:28.568581 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:28.568636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260810 14:39:29.192669 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:29.192208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:56469)
0810 14:39:29.192494 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:29.192559 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20260810 14:39:29.354570 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:29.354098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:41363)
0810 14:39:29.354404 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:29.354468 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260810 14:39:29.570214 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:29.569658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:40733)
0810 14:39:29.570024 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:29.570101 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
I20260810 14:39:30.194147 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:30.193670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.65:55747)
0810 14:39:30.193977 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:30.194039 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260810 14:39:30.356161 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:30.355702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.66:52597)
0810 14:39:30.355987 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:30.356054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260810 14:39:30.558893 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 7450
I20260810 14:39:30.568634 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 7642
I20260810 14:39:30.572293 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:30.571468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:38751 (local address 127.21.43.67:53115)
0810 14:39:30.572123 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:30.572192 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.126:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":107,"threads_started":1}
I20260810 14:39:30.578117 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 7794
2026-08-10T14:39:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59510 ms)
[----------] 30 tests from SecurityITest (252796 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:30 dist-test-slave-chsk krb5kdc[8006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:30 dist-test-slave-chsk krb5kdc[8006](info): set up 2 sockets
Aug 10 14:39:30 dist-test-slave-chsk krb5kdc[8006](info): commencing operation
krb5kdc: starting...
W20260810 14:39:31.683916 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:31 dist-test-slave-chsk krb5kdc[8006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:39:31.827288 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:44213
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:44213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:31.945834 8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:31.946180 8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:31.946262 8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:31.950624 8020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:31.950747 8020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:31.950784 8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:31.950824 8020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:31.950909 8020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:31.957680 8020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:44213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:31.958923 8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:31.960100 8020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:39:31.969666 8020 server_base.cc:1034] running on GCE node
W20260810 14:39:31.969599 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:31.969599 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:31.969590 8031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:31.970585 8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:31.971756 8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:31.972954 8020 hybrid_clock.cc:648] HybridClock initialized: now 1786372771972920 us; error 48 us; skew 500 ppm
Aug 10 14:39:31 dist-test-slave-chsk krb5kdc[8006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372771, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:31.975852 8020 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:39:31.976751 8020 webserver.cc:460] Webserver started at http://127.21.43.126:43301/ using document root <none> and password file <none>
I20260810 14:39:31.977325 8020 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:31.977416 8020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:31.977656 8020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:31.979784 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "c0cbdd47ad974153af12f5055bbe2899"
format_stamp: "Formatted at 2026-08-10 14:39:31 on dist-test-slave-chsk"
server_key: "5e450a67eaaf6c0f3955ecbcf25dea9a"
server_key_iv: "48b4d815ad248bf71b973162e86ad75e"
server_key_version: "encryptionkey@0"
I20260810 14:39:31.980353 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "c0cbdd47ad974153af12f5055bbe2899"
format_stamp: "Formatted at 2026-08-10 14:39:31 on dist-test-slave-chsk"
server_key: "5e450a67eaaf6c0f3955ecbcf25dea9a"
server_key_iv: "48b4d815ad248bf71b973162e86ad75e"
server_key_version: "encryptionkey@0"
I20260810 14:39:31.984177 8020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:39:31.986593 8038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:31.987855 8020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:39:31.988006 8020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "c0cbdd47ad974153af12f5055bbe2899"
format_stamp: "Formatted at 2026-08-10 14:39:31 on dist-test-slave-chsk"
server_key: "5e450a67eaaf6c0f3955ecbcf25dea9a"
server_key_iv: "48b4d815ad248bf71b973162e86ad75e"
server_key_version: "encryptionkey@0"
I20260810 14:39:31.988127 8020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:32.016198 8020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:32.019534 8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:32.019763 8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:32.027644 8020 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:44213
I20260810 14:39:32.027751 8096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:44213 every 8 connection(s)
I20260810 14:39:32.028703 8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:32.031822 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:32.035439 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8020
I20260810 14:39:32.035601 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:32.035907 21677 external_mini_cluster.cc:1428] Setting key 746f204dc0854625137fc696d877c0b0
I20260810 14:39:32.038275 8097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: Bootstrap starting.
I20260810 14:39:32.041155 8097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:32.042099 8097 log.cc:824] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:32.044482 8097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: No bootstrap required, opened a new log
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:32.048255 8097 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } }
I20260810 14:39:32.048525 8097 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:32.048602 8097 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cbdd47ad974153af12f5055bbe2899, State: Initialized, Role: FOLLOWER
I20260810 14:39:32.049103 8097 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } }
I20260810 14:39:32.049275 8097 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:32.049346 8097 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:32.049464 8097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:32.050686 8097 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } }
I20260810 14:39:32.051045 8097 leader_election.cc:302] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0cbdd47ad974153af12f5055bbe2899; no voters:
I20260810 14:39:32.051548 8097 leader_election.cc:288] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:32.051664 8102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:32.051944 8102 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [term 1 LEADER]: Becoming Leader. State: Replica: c0cbdd47ad974153af12f5055bbe2899, State: Running, Role: LEADER
I20260810 14:39:32.052342 8102 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } }
I20260810 14:39:32.052871 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.037402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.21.43.126:44213)
0810 14:39:32.037880 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.037900 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.037946 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:39:32.038789 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.038813 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.039258 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.039638 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:32.040382 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.041913 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.043067 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.043785 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.043957 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.046884 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:32.046890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:32.046903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:32.046932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:32.049475 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.050033 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.050039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.050043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.050182 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.050509 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.050512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.050513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.051001 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:32.051247 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.051540 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.051876 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":126,"threads_started":1}
I20260810 14:39:32.052917 8097 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:32.053601 8102 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0cbdd47ad974153af12f5055bbe2899. Latest consensus state: current_term: 1 leader_uuid: "c0cbdd47ad974153af12f5055bbe2899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } } }
I20260810 14:39:32.055552 8102 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:32.055755 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0cbdd47ad974153af12f5055bbe2899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0cbdd47ad974153af12f5055bbe2899" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 44213 } } }
I20260810 14:39:32.055915 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:32.056015 8120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:32.059867 8120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:32.065742 8120 catalog_manager.cc:1347] Generated new cluster ID: 292738f170414e36b0f1277e408a3cf5
I20260810 14:39:32.065837 8120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:32.102449 8120 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:32.103590 8120 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:32.119426 8120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: Generated new TSK 0
I20260810 14:39:32.120255 8120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:39:32.165059 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:32.281261 8127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:32.281638 8127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:32.281713 8127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:32.285972 8127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:32.286062 8127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:32.286157 8127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:32.291951 8127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:32.293159 8127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:32.294248 8127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:32.304534 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.304600 8127 server_base.cc:1034] running on GCE node
W20260810 14:39:32.304500 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:32.304514 8137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.305284 8127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:32.305960 8127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:32.307183 8127 hybrid_clock.cc:648] HybridClock initialized: now 1786372772307129 us; error 30 us; skew 500 ppm
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:32.309963 8127 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:39:32.310868 8127 webserver.cc:460] Webserver started at http://127.21.43.65:44275/ using document root <none> and password file <none>
I20260810 14:39:32.311535 8127 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:32.311628 8127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:32.311864 8127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:32.313956 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "eb9ce82983824d679998d90a2bc2ad7d"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "b917f78524448959c63d8edd0bd155dd"
server_key_iv: "6672e130805230ac52e5bf41dfd7e08a"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.314577 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9ce82983824d679998d90a2bc2ad7d"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "b917f78524448959c63d8edd0bd155dd"
server_key_iv: "6672e130805230ac52e5bf41dfd7e08a"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.318588 8127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:32.321028 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.322273 8127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:32.322402 8127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "eb9ce82983824d679998d90a2bc2ad7d"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "b917f78524448959c63d8edd0bd155dd"
server_key_iv: "6672e130805230ac52e5bf41dfd7e08a"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.322510 8127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:32.350598 8127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:32.354063 8127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:32.354278 8127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:32.354990 8127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:32.355894 8127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:32.355966 8127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.356038 8127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:32.356084 8127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.365052 8127 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:40665
I20260810 14:39:32.365155 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:40665 every 8 connection(s)
I20260810 14:39:32.366041 8127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:32.373258 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8127
I20260810 14:39:32.373412 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:32.373770 21677 external_mini_cluster.cc:1428] Setting key 933dddaf0e6ea373ec17a4f721fb7ff7
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:32.380954 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.367970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:57783 (local address 127.21.43.126:44213)
0810 14:39:32.368119 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.368123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.368971 (+ 848us) server_negotiation.cc:407] Connection header received
0810 14:39:32.370391 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.370395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.370457 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.370566 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:32.371817 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.372359 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.373805 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.374073 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.374189 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.376903 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:32.376906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:32.376908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:32.376932 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:32.378427 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.379085 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.379088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.379089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.379225 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.379627 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.379629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.379630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.379810 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:32.379888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.380636 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.380785 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:39:32.382252 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.368341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:44213 (local address 127.21.43.65:57783)
0810 14:39:32.368816 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.368861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.370170 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.370735 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.370741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.371117 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:32.371671 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.371685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.372491 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.372495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.373676 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.373690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.374247 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.374250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.374395 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.375013 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:32.375034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:32.376730 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:32.378550 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.378556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.378573 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.378960 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.379322 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.379328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.379332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.379534 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.379914 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:32.379924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:32.380314 (+ 390us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.380557 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.380995 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260810 14:39:32.383522 8276 heartbeater.cc:347] Connected to a master server at 127.21.43.126:44213
I20260810 14:39:32.383785 8276 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:32.384305 8276 heartbeater.cc:511] Master 127.21.43.126:44213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:39:32.385840 8057 ts_manager.cc:194] Registered new tserver with Master: eb9ce82983824d679998d90a2bc2ad7d (127.21.43.65:40665)
I20260810 14:39:32.386986 8057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:57783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:39:32.405916 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:32.521240 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:32.521544 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:32.521618 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:32.525684 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:32.525769 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:32.525875 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:32.531282 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:32.532367 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:32.533421 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:32.543299 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:32.543315 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.543339 8283 server_base.cc:1034] running on GCE node
W20260810 14:39:32.543577 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.543974 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:32.544642 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:32.545828 8283 hybrid_clock.cc:648] HybridClock initialized: now 1786372772545796 us; error 44 us; skew 500 ppm
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:32.548686 8283 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:39:32.549599 8283 webserver.cc:460] Webserver started at http://127.21.43.66:35391/ using document root <none> and password file <none>
I20260810 14:39:32.550163 8283 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:32.550251 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:32.550484 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:32.552600 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "f576352a737747a1a15b76d4436b3e24"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "1cf813cd530bc8946765ca7136aac640"
server_key_iv: "b848ac1b6c567a87b79af63cb9ab2ff2"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.553159 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "f576352a737747a1a15b76d4436b3e24"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "1cf813cd530bc8946765ca7136aac640"
server_key_iv: "b848ac1b6c567a87b79af63cb9ab2ff2"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.557101 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:39:32.559479 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.560685 8283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:39:32.560819 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "f576352a737747a1a15b76d4436b3e24"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "1cf813cd530bc8946765ca7136aac640"
server_key_iv: "b848ac1b6c567a87b79af63cb9ab2ff2"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.560928 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:32.583663 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:32.586936 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:32.587169 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:32.587769 8283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:32.588584 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:32.588649 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.588717 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:32.588764 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.597978 8283 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:38757
I20260810 14:39:32.598006 8421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:38757 every 8 connection(s)
I20260810 14:39:32.598871 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:32.603787 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8283
I20260810 14:39:32.603926 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:32.604247 21677 external_mini_cluster.cc:1428] Setting key 36d239e77921e2be4d4fe05b1c80ec6a
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:32.614173 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.600720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:51689 (local address 127.21.43.126:44213)
0810 14:39:32.600869 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.600873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.601731 (+ 858us) server_negotiation.cc:407] Connection header received
0810 14:39:32.603109 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.603113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.603194 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.603293 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:32.604843 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.605368 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.607033 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.607319 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.607404 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.610441 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:32.610444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:32.610446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:32.610470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:32.611918 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.612585 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.612587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.612589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.612658 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.613070 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.613072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.613073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.613241 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:32.613310 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.613950 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.614050 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:39:32.615463 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.601135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:44213 (local address 127.21.43.66:51689)
0810 14:39:32.601572 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.601623 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.602894 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.603467 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.603473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.604011 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:32.604681 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.604699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.605477 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.605480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.606892 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.606908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.607474 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.607481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.607728 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.608435 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:32.608451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:32.610272 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:32.612040 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.612048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.612068 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.612483 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.612762 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.612767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.612770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.612977 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.613360 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:32.613369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:32.613696 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.613925 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.614288 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:39:32.616744 8422 heartbeater.cc:347] Connected to a master server at 127.21.43.126:44213
I20260810 14:39:32.617002 8422 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:32.617463 8422 heartbeater.cc:511] Master 127.21.43.126:44213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:39:32.618732 8057 ts_manager.cc:194] Registered new tserver with Master: f576352a737747a1a15b76d4436b3e24 (127.21.43.66:38757)
I20260810 14:39:32.619482 8057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:51689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:39:32.634814 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:32.751279 8428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:32.751546 8428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:32.751618 8428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:32.755908 8428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:32.755975 8428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:32.756075 8428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:32.761467 8428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:44213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:32.762581 8428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:32.763720 8428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:32.773427 8435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:32.773428 8436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.773572 8428 server_base.cc:1034] running on GCE node
W20260810 14:39:32.773442 8439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:32.774116 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:32.774761 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:32.775954 8428 hybrid_clock.cc:648] HybridClock initialized: now 1786372772775938 us; error 27 us; skew 500 ppm
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:32.778767 8428 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:39:32.779645 8428 webserver.cc:460] Webserver started at http://127.21.43.67:43017/ using document root <none> and password file <none>
I20260810 14:39:32.780160 8428 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:32.780220 8428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:32.780385 8428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:32.782397 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "5cbc71f798c54db098c62040b60a16ea"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "07ecd1fda74e9450d3c26dda987acb59"
server_key_iv: "47fb5c3517a2d4c48eab4742c118f1e6"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.782912 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "5cbc71f798c54db098c62040b60a16ea"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "07ecd1fda74e9450d3c26dda987acb59"
server_key_iv: "47fb5c3517a2d4c48eab4742c118f1e6"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.786545 8428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:32.788904 8446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.790093 8428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:32.790212 8428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "5cbc71f798c54db098c62040b60a16ea"
format_stamp: "Formatted at 2026-08-10 14:39:32 on dist-test-slave-chsk"
server_key: "07ecd1fda74e9450d3c26dda987acb59"
server_key_iv: "47fb5c3517a2d4c48eab4742c118f1e6"
server_key_version: "encryptionkey@0"
I20260810 14:39:32.790315 8428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:32.817927 8428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:32.821236 8428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:32.821462 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:32.822158 8428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:32.823002 8428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:32.823050 8428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.823122 8428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:32.823201 8428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:32.831991 8428 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:34385
I20260810 14:39:32.832067 8579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:34385 every 8 connection(s)
I20260810 14:39:32.832969 8428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:39:32 dist-test-slave-chsk krb5kdc[8006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372772, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:32.842356 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8428
I20260810 14:39:32.842501 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:32.842793 21677 external_mini_cluster.cc:1428] Setting key 2dc6fbd78d64be7af9e847f0b250e173
I20260810 14:39:32.846686 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.834775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:55079 (local address 127.21.43.126:44213)
0810 14:39:32.834926 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.834929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.835775 (+ 846us) server_negotiation.cc:407] Connection header received
0810 14:39:32.837091 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.837095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.837155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.837255 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:32.838365 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.838894 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.840088 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.840299 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.840402 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.842886 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:32.842890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:32.842893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:32.842922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:32.844687 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.845298 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.845301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.845302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.845384 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.845705 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.845708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.845709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.845911 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:32.846005 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.846458 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.846566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:39:32.847625 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.835190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:44213 (local address 127.21.43.67:55079)
0810 14:39:32.835633 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.835673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.836886 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.837423 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.837429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.837770 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:32.838225 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.838237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.839031 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.839034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.839952 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.839961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.840456 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.840460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.840619 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.841157 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:32.841172 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:32.842702 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:32.844800 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.844805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.844816 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.845208 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.845465 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:32.845467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:32.845468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:32.845601 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:32.846008 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:32.846012 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:32.846257 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.846411 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.846660 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
I20260810 14:39:32.848630 8580 heartbeater.cc:347] Connected to a master server at 127.21.43.126:44213
I20260810 14:39:32.848870 8580 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:32.849393 8580 heartbeater.cc:511] Master 127.21.43.126:44213 requested a full tablet report, sending...
I20260810 14:39:32.850500 8057 ts_manager.cc:194] Registered new tserver with Master: 5cbc71f798c54db098c62040b60a16ea (127.21.43.67:34385)
I20260810 14:39:32.851048 8057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:55079
I20260810 14:39:32.856585 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:32.866097 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.858583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.21.43.126:44213)
0810 14:39:32.858729 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.858735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.858901 (+ 166us) server_negotiation.cc:407] Connection header received
0810 14:39:32.859046 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.859049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.859095 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.859204 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:32.859908 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.860429 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.861281 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.861495 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.861603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.862930 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:32.862933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:32.862936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:32.862965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:32.864389 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.864806 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.864810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.864812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.864925 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:32.865199 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:32.865203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:32.865205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:32.865425 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:32.865568 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.865803 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.865919 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:39:32.868925 8057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:32.871903 8057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:32.886636 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.880446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.21.43.66:38757)
0810 14:39:32.881160 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.881175 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.881202 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:39:32.881282 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.881287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.881494 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.881670 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:32.882182 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.883293 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.884838 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.885689 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.885798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.885942 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.886355 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.886455 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":148,"threads_started":1}
I20260810 14:39:32.886703 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.880493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:40665 (local address 127.0.0.1:46042)
0810 14:39:32.881432 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.881445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.881531 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.882517 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.882520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.882535 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:32.882649 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.882834 (+ 185us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.884188 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.884192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.885276 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.885287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.886284 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.886288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.886394 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.886418 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.886499 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.886570 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":47,"thread_start_us":110,"threads_started":1}
I20260810 14:39:32.886801 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.881277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.21.43.65:40665)
0810 14:39:32.882095 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.882113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.882140 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:39:32.882218 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.882222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.882405 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.882546 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:32.882964 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.884054 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.885417 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.886149 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.886255 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.886382 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.886543 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.886648 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":72,"threads_started":1}
I20260810 14:39:32.886550 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.880289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:38757 (local address 127.0.0.1:41440)
0810 14:39:32.880838 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.880887 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.881019 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.881889 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.881895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.881921 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:32.882063 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.882070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.883434 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.883437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.884716 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.884725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.885845 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.885849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.886052 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.886107 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.886239 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.886362 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":57,"thread_start_us":92,"threads_started":1}
I20260810 14:39:32.888053 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.881013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:34385 (local address 127.0.0.1:34368)
0810 14:39:32.882295 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.882308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.882397 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.882822 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.882828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.882849 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:32.882946 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.882952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.884190 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.884194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.886605 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.886613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.887667 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.887670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.887753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.887775 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.887856 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.887925 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":31,"thread_start_us":102,"threads_started":1}
I20260810 14:39:32.888114 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.881611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.21.43.67:34385)
0810 14:39:32.881959 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.881968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.882398 (+ 430us) server_negotiation.cc:407] Connection header received
0810 14:39:32.882502 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.882507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.882675 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.882817 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:32.883058 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.884031 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.886741 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.887546 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.887654 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.887782 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.887884 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.887969 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":161,"threads_started":1}
I20260810 14:39:32.889955 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be (DEFAULT_TABLE table=test-table [id=08245f2b6f1348df87400a9c985c4ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:32.890347 8504 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be (DEFAULT_TABLE table=test-table [id=08245f2b6f1348df87400a9c985c4ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:32.890399 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be (DEFAULT_TABLE table=test-table [id=08245f2b6f1348df87400a9c985c4ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:32.891228 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a97e9a6e8468886a1ae1748a814be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:32.891358 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a97e9a6e8468886a1ae1748a814be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:32.891356 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6a97e9a6e8468886a1ae1748a814be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:32.897799 8598 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Bootstrap starting.
I20260810 14:39:32.899315 8599 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Bootstrap starting.
I20260810 14:39:32.899447 8601 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: Bootstrap starting.
I20260810 14:39:32.900302 8598 tablet_bootstrap.cc:654] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:32.901539 8598 log.cc:824] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:32.901634 8601 tablet_bootstrap.cc:654] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:32.902120 8599 tablet_bootstrap.cc:654] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:32.902591 8601 log.cc:824] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:32.902930 8599 log.cc:824] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:32.903723 8598 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: No bootstrap required, opened a new log
I20260810 14:39:32.903928 8598 ts_tablet_manager.cc:1397] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:39:32.904563 8601 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: No bootstrap required, opened a new log
I20260810 14:39:32.904762 8601 ts_tablet_manager.cc:1397] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 14:39:32.905838 8599 tablet_bootstrap.cc:492] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: No bootstrap required, opened a new log
I20260810 14:39:32.906080 8599 ts_tablet_manager.cc:1397] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 14:39:32.908044 8598 raft_consensus.cc:348] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.908375 8598 raft_consensus.cc:374] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:32.908476 8598 raft_consensus.cc:729] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbc71f798c54db098c62040b60a16ea, State: Initialized, Role: FOLLOWER
I20260810 14:39:32.908806 8601 raft_consensus.cc:348] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.909039 8601 raft_consensus.cc:374] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:32.908988 8598 consensus_queue.cc:260] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.909112 8601 raft_consensus.cc:729] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9ce82983824d679998d90a2bc2ad7d, State: Initialized, Role: FOLLOWER
I20260810 14:39:32.909559 8601 consensus_queue.cc:260] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.909765 8580 heartbeater.cc:503] Master 127.21.43.126:44213 was elected leader, sending a full tablet report...
I20260810 14:39:32.910090 8598 ts_tablet_manager.cc:1428] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:39:32.909941 8599 raft_consensus.cc:348] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.910207 8599 raft_consensus.cc:374] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:32.910255 8599 raft_consensus.cc:729] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f576352a737747a1a15b76d4436b3e24, State: Initialized, Role: FOLLOWER
I20260810 14:39:32.910542 8601 ts_tablet_manager.cc:1428] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:39:32.910629 8599 consensus_queue.cc:260] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.910914 8276 heartbeater.cc:503] Master 127.21.43.126:44213 was elected leader, sending a full tablet report...
I20260810 14:39:32.911535 8599 ts_tablet_manager.cc:1428] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 14:39:32.912012 8422 heartbeater.cc:503] Master 127.21.43.126:44213 was elected leader, sending a full tablet report...
I20260810 14:39:32.923255 8605 raft_consensus.cc:482] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:32.923429 8605 raft_consensus.cc:504] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.924423 8605 leader_election.cc:288] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f576352a737747a1a15b76d4436b3e24 (127.21.43.66:38757), eb9ce82983824d679998d90a2bc2ad7d (127.21.43.65:40665)
I20260810 14:39:32.928376 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.924928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:49543 (local address 127.21.43.66:38757)
0810 14:39:32.925026 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.925030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.925054 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:39:32.925106 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.925108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.925148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.925238 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:32.925504 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.926069 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.927427 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.927829 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.927957 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.928010 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.928189 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.928255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:39:32.928377 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.924633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:38757 (local address 127.21.43.67:49543)
0810 14:39:32.924757 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.924773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.924865 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.925253 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.925257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.925275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:32.925394 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.925399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.926176 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.926181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.927307 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.927317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.927951 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.927955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.928059 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.928080 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.928174 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.928246 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:39:32.928493 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.924766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:34571 (local address 127.21.43.65:40665)
0810 14:39:32.924901 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.924906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.925243 (+ 337us) server_negotiation.cc:407] Connection header received
0810 14:39:32.925369 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.925372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.925415 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.925489 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:32.925761 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.926309 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.927548 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.927953 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.928082 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.928134 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.928303 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.928370 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:39:32.928541 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.924650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:40665 (local address 127.21.43.67:34571)
0810 14:39:32.925146 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.925159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:32.925262 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:32.925509 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:32.925512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:32.925527 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:32.925661 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.925666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.926411 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.926416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.927425 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:32.927434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.928071 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.928075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:32.928177 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.928198 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:39:32.928288 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:39:32.928355 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":55,"thread_start_us":140,"threads_started":1}
I20260810 14:39:32.929127 8221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a97e9a6e8468886a1ae1748a814be" candidate_uuid: "5cbc71f798c54db098c62040b60a16ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9ce82983824d679998d90a2bc2ad7d" is_pre_election: true
I20260810 14:39:32.929420 8221 raft_consensus.cc:2454] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbc71f798c54db098c62040b60a16ea in term 0.
I20260810 14:39:32.929557 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a97e9a6e8468886a1ae1748a814be" candidate_uuid: "5cbc71f798c54db098c62040b60a16ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576352a737747a1a15b76d4436b3e24" is_pre_election: true
I20260810 14:39:32.929841 8371 raft_consensus.cc:2454] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cbc71f798c54db098c62040b60a16ea in term 0.
I20260810 14:39:32.929934 8448 leader_election.cc:302] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbc71f798c54db098c62040b60a16ea, eb9ce82983824d679998d90a2bc2ad7d; no voters:
I20260810 14:39:32.930223 8605 raft_consensus.cc:2781] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:32.930327 8605 raft_consensus.cc:482] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:32.930392 8605 raft_consensus.cc:3037] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:32.931596 8605 raft_consensus.cc:504] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.931967 8605 leader_election.cc:288] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [CANDIDATE]: Term 1 election: Requested vote from peers f576352a737747a1a15b76d4436b3e24 (127.21.43.66:38757), eb9ce82983824d679998d90a2bc2ad7d (127.21.43.65:40665)
I20260810 14:39:32.932272 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a97e9a6e8468886a1ae1748a814be" candidate_uuid: "5cbc71f798c54db098c62040b60a16ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f576352a737747a1a15b76d4436b3e24"
I20260810 14:39:32.932413 8371 raft_consensus.cc:3037] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:32.932410 8221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6a97e9a6e8468886a1ae1748a814be" candidate_uuid: "5cbc71f798c54db098c62040b60a16ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9ce82983824d679998d90a2bc2ad7d"
I20260810 14:39:32.932552 8221 raft_consensus.cc:3037] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:32.933496 8371 raft_consensus.cc:2454] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbc71f798c54db098c62040b60a16ea in term 1.
I20260810 14:39:32.933670 8221 raft_consensus.cc:2454] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cbc71f798c54db098c62040b60a16ea in term 1.
I20260810 14:39:32.933846 8449 leader_election.cc:302] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbc71f798c54db098c62040b60a16ea, f576352a737747a1a15b76d4436b3e24; no voters:
I20260810 14:39:32.934021 8605 raft_consensus.cc:2781] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:32.934232 8605 raft_consensus.cc:686] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 1 LEADER]: Becoming Leader. State: Replica: 5cbc71f798c54db098c62040b60a16ea, State: Running, Role: LEADER
I20260810 14:39:32.934564 8605 consensus_queue.cc:237] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } }
I20260810 14:39:32.937808 8056 catalog_manager.cc:5495] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cbc71f798c54db098c62040b60a16ea (127.21.43.67). New cstate: current_term: 1 leader_uuid: "5cbc71f798c54db098c62040b60a16ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cbc71f798c54db098c62040b60a16ea" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 34385 } health_report { overall_health: HEALTHY } } }
I20260810 14:39:32.953182 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.949713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.21.43.67:34385)
0810 14:39:32.949847 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:32.949851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:32.949875 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:39:32.949983 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:32.949986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:32.950040 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:32.950130 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:32.950394 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.950915 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.951856 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:32.952076 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:32.952171 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:32.952428 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:32.952850 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:32.952942 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:32.952995 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 14:39:32.953049 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:39:32.960553 8371 raft_consensus.cc:1261] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 1 FOLLOWER]: Refusing update from remote peer 5cbc71f798c54db098c62040b60a16ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:32.960553 8221 raft_consensus.cc:1261] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 1 FOLLOWER]: Refusing update from remote peer 5cbc71f798c54db098c62040b60a16ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:32.961244 8605 consensus_queue.cc:1035] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9ce82983824d679998d90a2bc2ad7d" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:32.961450 8610 consensus_queue.cc:1035] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f576352a737747a1a15b76d4436b3e24" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 38757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:32.970932 8613 mvcc.cc:205] Tried to move back new op lower bound from 7316982878038921216 to 7316982877943066624. Current Snapshot: MvccSnapshot[applied={T|T < 7316982878038921216 or (T in {7316982878038921216})}]
I20260810 14:39:32.976787 8054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:32.976976 8054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:32.979564 8054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0cbdd47ad974153af12f5055bbe2899: Sending DeleteTablet for 3 replicas of tablet 5e6a97e9a6e8468886a1ae1748a814be
I20260810 14:39:32.980433 8199 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:46042
I20260810 14:39:32.980453 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:41440
I20260810 14:39:32.980741 8504 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6a97e9a6e8468886a1ae1748a814be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:32 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:34368
I20260810 14:39:32.980969 8623 tablet_replica.cc:330] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d: stopping tablet replica
I20260810 14:39:32.981228 8627 tablet_replica.cc:330] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: stopping tablet replica
I20260810 14:39:32.981340 8623 raft_consensus.cc:2229] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:32.981596 8627 raft_consensus.cc:2229] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:39:32.981705 8623 raft_consensus.cc:2258] T 5e6a97e9a6e8468886a1ae1748a814be P eb9ce82983824d679998d90a2bc2ad7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:32.982105 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8127
I20260810 14:39:32.982229 8626 tablet_replica.cc:330] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: stopping tablet replica
I20260810 14:39:32.982510 8626 raft_consensus.cc:2229] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:32.982666 8627 raft_consensus.cc:2258] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:32.982774 8626 raft_consensus.cc:2258] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:32.984673 8626 ts_tablet_manager.cc:1905] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:32.984671 8627 ts_tablet_manager.cc:1905] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:32.987433 8627 ts_tablet_manager.cc:1918] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:32.987433 8626 ts_tablet_manager.cc:1918] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:32.987581 8627 log.cc:1196] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/5e6a97e9a6e8468886a1ae1748a814be
I20260810 14:39:32.987591 8626 log.cc:1196] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/5e6a97e9a6e8468886a1ae1748a814be
I20260810 14:39:32.987984 8626 ts_tablet_manager.cc:1939] T 5e6a97e9a6e8468886a1ae1748a814be P f576352a737747a1a15b76d4436b3e24: Deleting consensus metadata
I20260810 14:39:32.988009 8627 ts_tablet_manager.cc:1939] T 5e6a97e9a6e8468886a1ae1748a814be P 5cbc71f798c54db098c62040b60a16ea: Deleting consensus metadata
I20260810 14:39:32.991425 8041 catalog_manager.cc:4841] TS 5cbc71f798c54db098c62040b60a16ea (127.21.43.67:34385): tablet 5e6a97e9a6e8468886a1ae1748a814be (table test-table [id=08245f2b6f1348df87400a9c985c4ce5]) successfully deleted
I20260810 14:39:32.991642 8041 catalog_manager.cc:4841] TS f576352a737747a1a15b76d4436b3e24 (127.21.43.66:38757): tablet 5e6a97e9a6e8468886a1ae1748a814be (table test-table [id=08245f2b6f1348df87400a9c985c4ce5]) successfully deleted
W20260810 14:39:32.995659 8040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:40665 (error 108)
I20260810 14:39:32.995951 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8283
I20260810 14:39:32.997498 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:32.997215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:40665 (local address 127.0.0.1:46044)
0810 14:39:32.997352 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:32.997420 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:40665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:39:32.997641 8040 catalog_manager.cc:4584] TS eb9ce82983824d679998d90a2bc2ad7d (127.21.43.65:40665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e6a97e9a6e8468886a1ae1748a814be: Network error: Client connection negotiation failed: client connection to 127.21.43.65:40665: connect: Connection refused (error 111)
I20260810 14:39:33.007437 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8428
I20260810 14:39:33.016501 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8020
2026-08-10T14:39:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2437 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T14:39:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:33Z Disabled control of system clock
I20260810 14:39:33.051213 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40839
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:33.164996 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:33.165287 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:33.165324 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:33.169273 8634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:33.169334 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:33.169350 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:33.169366 8634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:33.169382 8634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:33.174121 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:33.175237 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:33.176626 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:39:33.185226 8634 server_base.cc:1034] running on GCE node
W20260810 14:39:33.185235 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.185173 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.185457 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.186038 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:33.187265 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:33.188441 8634 hybrid_clock.cc:648] HybridClock initialized: now 1786372773188413 us; error 40 us; skew 500 ppm
I20260810 14:39:33.190342 8634 webserver.cc:460] Webserver started at http://127.21.43.126:41301/ using document root <none> and password file <none>
I20260810 14:39:33.190992 8634 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:33.191104 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:33.191375 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:33.193518 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "3a87840d7b5d4a6ea1af68bd491a48cd"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "74f906f6c86d48bad0a671a93aee3f18"
server_key_iv: "846c63f3bd9db40eeb7228c474dac52b"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.194113 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "3a87840d7b5d4a6ea1af68bd491a48cd"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "74f906f6c86d48bad0a671a93aee3f18"
server_key_iv: "846c63f3bd9db40eeb7228c474dac52b"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.197903 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 14:39:33.200320 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.201512 8634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:33.201660 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "3a87840d7b5d4a6ea1af68bd491a48cd"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "74f906f6c86d48bad0a671a93aee3f18"
server_key_iv: "846c63f3bd9db40eeb7228c474dac52b"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.201771 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:33.217023 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:33.217757 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:33.217913 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:33.224759 8634 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40839
I20260810 14:39:33.224772 8706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40839 every 8 connection(s)
I20260810 14:39:33.225718 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:33.227488 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8634
I20260810 14:39:33.227625 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:33.227907 21677 external_mini_cluster.cc:1428] Setting key 5ed32cdce2476290fa8c5b8310c41532
I20260810 14:39:33.229641 8707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:33.235270 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: Bootstrap starting.
I20260810 14:39:33.237010 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.229386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.21.43.126:40839)
0810 14:39:33.230084 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.230101 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.230136 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:39:33.230776 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.230791 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.231233 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.231556 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:33.231973 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.232922 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.233810 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.234270 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.234440 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.235364 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:33.235373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:33.235390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:33.235781 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:33.235836 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:33.235856 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:33.235965 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.236088 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.236336 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":146,"threads_started":1}
I20260810 14:39:33.237676 8707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:33.238585 8707 log.cc:824] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:33.240650 8707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: No bootstrap required, opened a new log
I20260810 14:39:33.244449 8707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } }
I20260810 14:39:33.244688 8707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:33.244725 8707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a87840d7b5d4a6ea1af68bd491a48cd, State: Initialized, Role: FOLLOWER
I20260810 14:39:33.245085 8707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } }
I20260810 14:39:33.245198 8707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:33.245237 8707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:33.245301 8707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:33.246176 8707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } }
I20260810 14:39:33.246475 8707 leader_election.cc:302] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a87840d7b5d4a6ea1af68bd491a48cd; no voters:
I20260810 14:39:33.246748 8707 leader_election.cc:288] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:33.246873 8713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:33.247128 8713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [term 1 LEADER]: Becoming Leader. State: Replica: 3a87840d7b5d4a6ea1af68bd491a48cd, State: Running, Role: LEADER
I20260810 14:39:33.247529 8713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } }
I20260810 14:39:33.248112 8707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:33.249189 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a87840d7b5d4a6ea1af68bd491a48cd. Latest consensus state: current_term: 1 leader_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } } }
I20260810 14:39:33.249326 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:33.249635 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a87840d7b5d4a6ea1af68bd491a48cd" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40839 } } }
I20260810 14:39:33.249758 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:33.250146 8723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:33.253135 8723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:33.258653 8723 catalog_manager.cc:1347] Generated new cluster ID: c6dd2c600aaa4966926070f281c7afb0
I20260810 14:39:33.258790 8723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:33.277290 8723 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:33.278625 8723 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:33.290212 8723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: Generated new TSK 0
I20260810 14:39:33.291038 8723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:33.302537 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--builtin_ntp_servers=127.21.43.84:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:33.414253 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:33.414552 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:33.414620 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:33.418709 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:33.418781 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:33.418869 8733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:33.423794 8733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:33.424844 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:33.426213 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:33.435465 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.435604 8733 server_base.cc:1034] running on GCE node
W20260810 14:39:33.435456 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.435460 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.436177 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:33.436822 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:33.438048 8733 hybrid_clock.cc:648] HybridClock initialized: now 1786372773438017 us; error 46 us; skew 500 ppm
I20260810 14:39:33.439965 8733 webserver.cc:460] Webserver started at http://127.21.43.65:34087/ using document root <none> and password file <none>
I20260810 14:39:33.440547 8733 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:33.440611 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:33.440862 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:33.442899 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "2e6d064e084c4de885c2854edbd74e17"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c8f673662f1bc45ad837b2c7d43045dc"
server_key_iv: "f4bd2a010f88743e1a08ce81900b2961"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.443514 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "2e6d064e084c4de885c2854edbd74e17"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c8f673662f1bc45ad837b2c7d43045dc"
server_key_iv: "f4bd2a010f88743e1a08ce81900b2961"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.447125 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:33.449762 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.450950 8733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:39:33.451081 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "2e6d064e084c4de885c2854edbd74e17"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c8f673662f1bc45ad837b2c7d43045dc"
server_key_iv: "f4bd2a010f88743e1a08ce81900b2961"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.451234 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:33.464846 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:33.465456 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:33.465646 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:33.466187 8733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:33.467082 8733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:33.467147 8733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.467254 8733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:33.467300 8733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.476373 8733 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:38137
I20260810 14:39:33.476433 8874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:38137 every 8 connection(s)
I20260810 14:39:33.477284 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:33.480072 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8733
I20260810 14:39:33.480201 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:33.480551 21677 external_mini_cluster.cc:1428] Setting key e2dc594c0531ee70f21d98edfe1a6ff6
I20260810 14:39:33.482585 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--builtin_ntp_servers=127.21.43.84:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:33.487509 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.479131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:44699 (local address 127.21.43.126:40839)
0810 14:39:33.479284 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.479288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.480532 (+ 1244us) server_negotiation.cc:407] Connection header received
0810 14:39:33.482035 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.482038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.482088 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.482158 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:33.483348 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.483873 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.485238 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.485634 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.485756 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.486856 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:33.486863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:33.486868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:33.486889 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:33.486915 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:33.486930 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:33.487017 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.487180 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.487281 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:39:33.488256 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.479531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40839 (local address 127.21.43.65:44699)
0810 14:39:33.480350 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.480408 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.481801 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.482336 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.482347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.482459 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:33.483197 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.483221 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.484001 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.484007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.485092 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.485102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.485805 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.485808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.485949 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.486586 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:33.486587 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:33.486628 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:33.486692 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:33.487023 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:33.487030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:33.487035 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.487127 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.487425 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":117,"threads_started":1}
I20260810 14:39:33.489513 8875 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40839
I20260810 14:39:33.489738 8875 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:33.490252 8875 heartbeater.cc:511] Master 127.21.43.126:40839 requested a full tablet report, sending...
I20260810 14:39:33.491794 8665 ts_manager.cc:194] Registered new tserver with Master: 2e6d064e084c4de885c2854edbd74e17 (127.21.43.65:38137)
I20260810 14:39:33.493182 8665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.65:44699
W20260810 14:39:33.597651 8880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:33.597944 8880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:33.598011 8880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:33.602214 8880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:33.602298 8880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:33.602381 8880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:33.607241 8880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:33.608317 8880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:33.609719 8880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:33.619025 8885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.619045 8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.619060 8886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.619200 8880 server_base.cc:1034] running on GCE node
I20260810 14:39:33.619810 8880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:33.620507 8880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:33.621701 8880 hybrid_clock.cc:648] HybridClock initialized: now 1786372773621684 us; error 29 us; skew 500 ppm
I20260810 14:39:33.623632 8880 webserver.cc:460] Webserver started at http://127.21.43.66:33677/ using document root <none> and password file <none>
I20260810 14:39:33.624212 8880 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:33.624305 8880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:33.624537 8880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:33.626562 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "6851b16de099477c910cd7d259a13b9e"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "4e05b6ebe16682f7ab927897e00dc084"
server_key_iv: "64bcd921437ef3982504f6d4d5e2e23c"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.627130 8880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "6851b16de099477c910cd7d259a13b9e"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "4e05b6ebe16682f7ab927897e00dc084"
server_key_iv: "64bcd921437ef3982504f6d4d5e2e23c"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.630849 8880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:33.633435 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.634570 8880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:33.634699 8880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "6851b16de099477c910cd7d259a13b9e"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "4e05b6ebe16682f7ab927897e00dc084"
server_key_iv: "64bcd921437ef3982504f6d4d5e2e23c"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.634810 8880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:33.654228 8880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:33.654948 8880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:33.655194 8880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:33.655843 8880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:33.656771 8880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:33.656836 8880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.656904 8880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:33.656950 8880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.665608 8880 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:36877
I20260810 14:39:33.665656 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:36877 every 8 connection(s)
I20260810 14:39:33.666499 8880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:33.670310 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 8880
I20260810 14:39:33.670449 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:33.670727 21677 external_mini_cluster.cc:1428] Setting key 642f9cc1cb4ca8dd81b852bdca27eaae
I20260810 14:39:33.672783 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--builtin_ntp_servers=127.21.43.84:42657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:33.676332 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.668324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:44091 (local address 127.21.43.126:40839)
0810 14:39:33.668466 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.668469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.669444 (+ 975us) server_negotiation.cc:407] Connection header received
0810 14:39:33.670926 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.670930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.670991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.671099 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:33.672071 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.672807 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.674040 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.674439 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.674565 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.675889 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:33.675892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:33.675893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:33.675905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:33.675923 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:33.675931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:33.675990 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.676175 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.676223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:39:33.677489 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.668636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40839 (local address 127.21.43.66:44091)
0810 14:39:33.669286 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.669326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.670698 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.671301 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.671312 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.671416 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:33.671887 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.671900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.672962 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.672968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.673912 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.673922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.674659 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.674666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.674898 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.675584 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:33.675588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:33.675643 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:33.675731 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:33.676016 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:33.676028 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:33.676036 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.676160 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.676449 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260810 14:39:33.678664 9031 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40839
I20260810 14:39:33.678890 9031 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:33.679440 9031 heartbeater.cc:511] Master 127.21.43.126:40839 requested a full tablet report, sending...
I20260810 14:39:33.680497 8665 ts_manager.cc:194] Registered new tserver with Master: 6851b16de099477c910cd7d259a13b9e (127.21.43.66:36877)
I20260810 14:39:33.681384 8665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.66:44091
W20260810 14:39:33.794319 9034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:33.794600 9034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:33.794639 9034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:33.798579 9034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:33.798636 9034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:33.798691 9034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:33.803498 9034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:33.804623 9034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:33.806104 9034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:33.815815 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:33.815914 9039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.816004 9034 server_base.cc:1034] running on GCE node
W20260810 14:39:33.815811 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:33.816552 9034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:33.817196 9034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:33.818356 9034 hybrid_clock.cc:648] HybridClock initialized: now 1786372773818343 us; error 29 us; skew 500 ppm
I20260810 14:39:33.820407 9034 webserver.cc:460] Webserver started at http://127.21.43.67:39465/ using document root <none> and password file <none>
I20260810 14:39:33.821079 9034 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:33.821146 9034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:33.821384 9034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:33.823608 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "35b012efaeb442c3af4ea09b3d869ea4"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c84b55e9456a690f7cce4ac98afbdf5a"
server_key_iv: "4aa6f4332583f4232f3f1079effc5f83"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.824234 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "35b012efaeb442c3af4ea09b3d869ea4"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c84b55e9456a690f7cce4ac98afbdf5a"
server_key_iv: "4aa6f4332583f4232f3f1079effc5f83"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.828169 9034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:39:33.830863 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.832195 9034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:33.832341 9034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "35b012efaeb442c3af4ea09b3d869ea4"
format_stamp: "Formatted at 2026-08-10 14:39:33 on dist-test-slave-chsk"
server_key: "c84b55e9456a690f7cce4ac98afbdf5a"
server_key_iv: "4aa6f4332583f4232f3f1079effc5f83"
server_key_version: "encryptionkey@0"
I20260810 14:39:33.832464 9034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:33.845613 9034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:33.846398 9034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:33.846621 9034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:33.847352 9034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:33.848379 9034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:33.848452 9034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.848521 9034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:33.848555 9034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:33.857959 9034 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:36273
I20260810 14:39:33.857988 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:36273 every 8 connection(s)
I20260810 14:39:33.858924 9034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:33.861012 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9034
I20260810 14:39:33.861143 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:33.861435 21677 external_mini_cluster.cc:1428] Setting key e2617fc36f40432556e460e3a0d1f570
I20260810 14:39:33.868639 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.860851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:57553 (local address 127.21.43.126:40839)
0810 14:39:33.861027 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.861032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.862324 (+ 1292us) server_negotiation.cc:407] Connection header received
0810 14:39:33.863914 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.863918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.863975 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.864056 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:33.864932 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.865463 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.866612 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.866845 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.866955 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.868122 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:33.868125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:33.868128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:33.868141 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:33.868167 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:33.868175 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:33.868252 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.868411 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.868476 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:39:33.869537 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.861364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:40839 (local address 127.21.43.67:57553)
0810 14:39:33.862169 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.862225 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.863707 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.864222 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.864229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.864313 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:33.864788 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.864801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.865602 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.865605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.866482 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.866492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.867011 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.867016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.867213 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.867830 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:33.867831 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:33.867871 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:33.867936 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:33.868254 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:33.868266 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:33.868274 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.868401 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.868699 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260810 14:39:33.870656 9176 heartbeater.cc:347] Connected to a master server at 127.21.43.126:40839
I20260810 14:39:33.870905 9176 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:33.871500 9176 heartbeater.cc:511] Master 127.21.43.126:40839 requested a full tablet report, sending...
I20260810 14:39:33.872542 8665 ts_manager.cc:194] Registered new tserver with Master: 35b012efaeb442c3af4ea09b3d869ea4 (127.21.43.67:36273)
I20260810 14:39:33.873131 8665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.67:57553
I20260810 14:39:33.875439 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:33.881204 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.877415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.21.43.126:40839)
0810 14:39:33.877582 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.877586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.877729 (+ 143us) server_negotiation.cc:407] Connection header received
0810 14:39:33.877818 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.877821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.877862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.877927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:33.878173 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.878690 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.879558 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.879772 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.879858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.880749 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:33.880752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:33.880754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:33.880766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:33.880787 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:33.880794 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:33.880857 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.880954 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.881010 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:39:33.884385 8665 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:33.887559 8665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:33.902467 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.896714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36273 (local address 127.0.0.1:51152)
0810 14:39:33.897626 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.897643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.897756 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.898510 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.898515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.898540 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:33.898695 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.898717 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.899886 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.899891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.901108 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.901118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.901918 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.901922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.902043 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.902088 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.902185 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.902303 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"spinlock_wait_cycles":2944,"thread_start_us":160,"threads_started":1}
I20260810 14:39:33.902494 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.896977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.21.43.67:36273)
0810 14:39:33.897462 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.897479 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.897917 (+ 438us) server_negotiation.cc:407] Connection header received
0810 14:39:33.898103 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.898111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.898348 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.898528 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:33.898848 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.899771 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.901233 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.901791 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.901927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.902158 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.902227 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.902321 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":164,"threads_started":1}
I20260810 14:39:33.903229 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.896671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38137 (local address 127.0.0.1:50708)
0810 14:39:33.897548 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.897609 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.897757 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.898931 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.898934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.898976 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:33.899089 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.899096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.900618 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.900621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.901739 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.901746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.902832 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.902836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.902926 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.902952 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.903003 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.903068 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260810 14:39:33.903585 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.897425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:36877 (local address 127.0.0.1:41478)
0810 14:39:33.898165 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.898180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.898277 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.898760 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.898765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.898790 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:33.898900 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.898909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.900330 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.900334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.901730 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.901741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.903134 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.903186 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.903261 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.903297 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.903379 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.903448 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":141,"thread_start_us":111,"threads_started":1}
I20260810 14:39:33.903991 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.896782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.21.43.65:38137)
0810 14:39:33.898509 (+ 1727us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.898521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.898541 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:39:33.898610 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.898615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.898777 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.899276 (+ 499us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:33.899421 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.900496 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.901902 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.902691 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.903512 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.903701 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.903750 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.903838 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":130,"threads_started":1}
I20260810 14:39:33.904004 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.897864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.21.43.66:36877)
0810 14:39:33.898258 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.898267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.898286 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:39:33.898431 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.898437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.898610 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.898766 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:33.899017 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.900198 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.901875 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.902814 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.902918 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.903076 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.903775 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.903877 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":124,"threads_started":1}
I20260810 14:39:33.906037 9104 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 (DEFAULT_TABLE table=test-table [id=1633337e0a0e49a0801359b76868c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:33.906872 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 (DEFAULT_TABLE table=test-table [id=1633337e0a0e49a0801359b76868c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:33.906809 8801 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 (DEFAULT_TABLE table=test-table [id=1633337e0a0e49a0801359b76868c53b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:33.907325 9104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c46a14fc641119c41d3f7794339c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:33.907799 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c46a14fc641119c41d3f7794339c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:33.907835 8801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c46a14fc641119c41d3f7794339c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:33.914534 9194 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Bootstrap starting.
I20260810 14:39:33.916024 9196 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Bootstrap starting.
I20260810 14:39:33.916944 9194 tablet_bootstrap.cc:654] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:33.917794 9194 log.cc:824] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:33.917851 9196 tablet_bootstrap.cc:654] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:33.918520 9196 log.cc:824] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:33.918687 9197 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Bootstrap starting.
I20260810 14:39:33.920279 9194 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: No bootstrap required, opened a new log
I20260810 14:39:33.920512 9194 ts_tablet_manager.cc:1397] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:39:33.920727 9196 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: No bootstrap required, opened a new log
I20260810 14:39:33.920926 9196 ts_tablet_manager.cc:1397] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:39:33.921608 9197 tablet_bootstrap.cc:654] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:33.922489 9197 log.cc:824] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:33.924485 9194 raft_consensus.cc:348] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.924746 9194 raft_consensus.cc:374] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:33.924801 9194 raft_consensus.cc:729] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6851b16de099477c910cd7d259a13b9e, State: Initialized, Role: FOLLOWER
I20260810 14:39:33.924805 9197 tablet_bootstrap.cc:492] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: No bootstrap required, opened a new log
I20260810 14:39:33.924997 9197 ts_tablet_manager.cc:1397] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 14:39:33.925268 9194 consensus_queue.cc:260] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.925215 9196 raft_consensus.cc:348] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.925494 9196 raft_consensus.cc:374] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:33.925653 9196 raft_consensus.cc:729] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b012efaeb442c3af4ea09b3d869ea4, State: Initialized, Role: FOLLOWER
I20260810 14:39:33.926170 9196 consensus_queue.cc:260] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.926355 9194 ts_tablet_manager.cc:1428] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 14:39:33.926730 9031 heartbeater.cc:503] Master 127.21.43.126:40839 was elected leader, sending a full tablet report...
I20260810 14:39:33.927258 9196 ts_tablet_manager.cc:1428] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 14:39:33.927632 9176 heartbeater.cc:503] Master 127.21.43.126:40839 was elected leader, sending a full tablet report...
I20260810 14:39:33.930419 9197 raft_consensus.cc:348] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.930756 9197 raft_consensus.cc:374] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:33.930846 9197 raft_consensus.cc:729] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6d064e084c4de885c2854edbd74e17, State: Initialized, Role: FOLLOWER
I20260810 14:39:33.931347 9197 consensus_queue.cc:260] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.932099 8875 heartbeater.cc:503] Master 127.21.43.126:40839 was elected leader, sending a full tablet report...
I20260810 14:39:33.932528 9197 ts_tablet_manager.cc:1428] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 14:39:33.933591 9204 raft_consensus.cc:482] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:33.933763 9204 raft_consensus.cc:504] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.936218 9204 leader_election.cc:288] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b012efaeb442c3af4ea09b3d869ea4 (127.21.43.67:36273), 6851b16de099477c910cd7d259a13b9e (127.21.43.66:36877)
I20260810 14:39:33.940726 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.935104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:36877 (local address 127.21.43.65:53015)
0810 14:39:33.935286 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.935305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.935385 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.936036 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.936041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.936082 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:33.936210 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.936220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.937313 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.937317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.939282 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.939295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.940201 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.940205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.940367 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.940409 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.940499 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.940587 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":290}
I20260810 14:39:33.940739 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.935420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:53015 (local address 127.21.43.66:36877)
0810 14:39:33.935780 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.935784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.935799 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:39:33.935849 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.935853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.935908 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.935985 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:33.936335 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.937189 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.939420 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.940039 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.940176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.940227 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.940567 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.940641 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260810 14:39:33.940747 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.935569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:38603 (local address 127.21.43.67:36273)
0810 14:39:33.935722 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.935728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.935779 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:39:33.935949 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.935951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.936020 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.936094 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:33.936489 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.937208 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.939427 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.940048 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.940176 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.940227 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.940558 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.940634 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:39:33.941116 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.935104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:36273 (local address 127.21.43.65:38603)
0810 14:39:33.935686 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:33.935701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:33.935859 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:33.936191 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:33.936194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:33.936208 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:33.936343 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.936349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.937313 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.937317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.939282 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:33.939295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.940201 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.940205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:33.940362 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.940408 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 14:39:33.940952 (+ 544us) client_negotiation.cc:240] Negotiation successful
0810 14:39:33.941015 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":701,"thread_start_us":114,"threads_started":1}
I20260810 14:39:33.941941 9126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c46a14fc641119c41d3f7794339c4" candidate_uuid: "2e6d064e084c4de885c2854edbd74e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b012efaeb442c3af4ea09b3d869ea4" is_pre_election: true
I20260810 14:39:33.941941 8979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c46a14fc641119c41d3f7794339c4" candidate_uuid: "2e6d064e084c4de885c2854edbd74e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6851b16de099477c910cd7d259a13b9e" is_pre_election: true
I20260810 14:39:33.942307 9126 raft_consensus.cc:2454] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6d064e084c4de885c2854edbd74e17 in term 0.
I20260810 14:39:33.942307 8979 raft_consensus.cc:2454] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e6d064e084c4de885c2854edbd74e17 in term 0.
I20260810 14:39:33.942889 8752 leader_election.cc:302] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e6d064e084c4de885c2854edbd74e17, 35b012efaeb442c3af4ea09b3d869ea4, 6851b16de099477c910cd7d259a13b9e; no voters:
I20260810 14:39:33.943236 9204 raft_consensus.cc:2781] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:33.943333 9204 raft_consensus.cc:482] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:33.943379 9204 raft_consensus.cc:3037] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:33.944439 9204 raft_consensus.cc:504] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.944828 9204 leader_election.cc:288] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [CANDIDATE]: Term 1 election: Requested vote from peers 35b012efaeb442c3af4ea09b3d869ea4 (127.21.43.67:36273), 6851b16de099477c910cd7d259a13b9e (127.21.43.66:36877)
I20260810 14:39:33.945222 9126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c46a14fc641119c41d3f7794339c4" candidate_uuid: "2e6d064e084c4de885c2854edbd74e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b012efaeb442c3af4ea09b3d869ea4"
I20260810 14:39:33.945273 8979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c46a14fc641119c41d3f7794339c4" candidate_uuid: "2e6d064e084c4de885c2854edbd74e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6851b16de099477c910cd7d259a13b9e"
I20260810 14:39:33.945389 8979 raft_consensus.cc:3037] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:33.945371 9126 raft_consensus.cc:3037] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:33.946798 9126 raft_consensus.cc:2454] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6d064e084c4de885c2854edbd74e17 in term 1.
I20260810 14:39:33.946798 8979 raft_consensus.cc:2454] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6d064e084c4de885c2854edbd74e17 in term 1.
I20260810 14:39:33.947279 8752 leader_election.cc:302] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6d064e084c4de885c2854edbd74e17, 35b012efaeb442c3af4ea09b3d869ea4; no voters:
I20260810 14:39:33.947479 9204 raft_consensus.cc:2781] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:33.947764 9204 raft_consensus.cc:686] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [term 1 LEADER]: Becoming Leader. State: Replica: 2e6d064e084c4de885c2854edbd74e17, State: Running, Role: LEADER
I20260810 14:39:33.948173 9204 consensus_queue.cc:237] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } }
I20260810 14:39:33.951687 8665 catalog_manager.cc:5495] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e6d064e084c4de885c2854edbd74e17 (127.21.43.65). New cstate: current_term: 1 leader_uuid: "2e6d064e084c4de885c2854edbd74e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e6d064e084c4de885c2854edbd74e17" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 } health_report { overall_health: UNKNOWN } } }
I20260810 14:39:33.968255 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:33.964617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.21.43.65:38137)
0810 14:39:33.964765 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:33.964770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:33.964785 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:39:33.964926 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:33.964928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:33.964990 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:33.965100 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:33.965351 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.965862 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.966735 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:33.966955 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:33.967064 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:33.967418 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:33.967851 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:33.967987 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:33.968024 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 14:39:33.968080 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:39:33.974589 8979 raft_consensus.cc:1261] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 1 FOLLOWER]: Refusing update from remote peer 2e6d064e084c4de885c2854edbd74e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:33.974589 9126 raft_consensus.cc:1261] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2e6d064e084c4de885c2854edbd74e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:33.975484 9204 consensus_queue.cc:1035] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b012efaeb442c3af4ea09b3d869ea4" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:33.975704 9207 consensus_queue.cc:1035] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6851b16de099477c910cd7d259a13b9e" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:39:33.978433 8877 tablet.cc:2369] T 1d7c46a14fc641119c41d3f7794339c4 P 2e6d064e084c4de885c2854edbd74e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:39:33.982600 9211 mvcc.cc:205] Tried to move back new op lower bound from 7316982882192011264 to 7316982882094972928. Current Snapshot: MvccSnapshot[applied={T|T < 7316982882192011264}]
I20260810 14:39:33.984450 9212 mvcc.cc:205] Tried to move back new op lower bound from 7316982882192011264 to 7316982882094972928. Current Snapshot: MvccSnapshot[applied={T|T < 7316982882192011264}]
I20260810 14:39:33.992515 8663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:33.992718 8663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:33.995338 8663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a87840d7b5d4a6ea1af68bd491a48cd: Sending DeleteTablet for 3 replicas of tablet 1d7c46a14fc641119c41d3f7794339c4
I20260810 14:39:33.996250 8801 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:33 UTC) from {username='slave'} at 127.0.0.1:50708
I20260810 14:39:33.996294 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:33 UTC) from {username='slave'} at 127.0.0.1:41478
I20260810 14:39:33.996800 9104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c46a14fc641119c41d3f7794339c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:33 UTC) from {username='slave'} at 127.0.0.1:51152
I20260810 14:39:33.996949 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8733
I20260810 14:39:33.998545 9220 tablet_replica.cc:330] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: stopping tablet replica
I20260810 14:39:33.998672 9221 tablet_replica.cc:330] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: stopping tablet replica
I20260810 14:39:33.998957 9221 raft_consensus.cc:2229] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:33.998957 9220 raft_consensus.cc:2229] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:33.999354 9220 raft_consensus.cc:2258] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:33.999354 9221 raft_consensus.cc:2258] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:34.001294 9220 ts_tablet_manager.cc:1905] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:34.001749 9221 ts_tablet_manager.cc:1905] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:34.003767 9220 ts_tablet_manager.cc:1918] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:34.003888 9220 log.cc:1196] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/1d7c46a14fc641119c41d3f7794339c4
I20260810 14:39:34.004190 9220 ts_tablet_manager.cc:1939] T 1d7c46a14fc641119c41d3f7794339c4 P 6851b16de099477c910cd7d259a13b9e: Deleting consensus metadata
I20260810 14:39:34.005445 9221 ts_tablet_manager.cc:1918] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:34.005561 9221 log.cc:1196] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/1d7c46a14fc641119c41d3f7794339c4
I20260810 14:39:34.005666 8649 catalog_manager.cc:4841] TS 6851b16de099477c910cd7d259a13b9e (127.21.43.66:36877): tablet 1d7c46a14fc641119c41d3f7794339c4 (table test-table [id=1633337e0a0e49a0801359b76868c53b]) successfully deleted
I20260810 14:39:34.005865 9221 ts_tablet_manager.cc:1939] T 1d7c46a14fc641119c41d3f7794339c4 P 35b012efaeb442c3af4ea09b3d869ea4: Deleting consensus metadata
I20260810 14:39:34.007252 8651 catalog_manager.cc:4841] TS 35b012efaeb442c3af4ea09b3d869ea4 (127.21.43.67:36273): tablet 1d7c46a14fc641119c41d3f7794339c4 (table test-table [id=1633337e0a0e49a0801359b76868c53b]) successfully deleted
W20260810 14:39:34.008368 8652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:38137 (error 108)
I20260810 14:39:34.008628 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8880
I20260810 14:39:34.011364 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:34.010996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38137 (local address 127.0.0.1:50734)
0810 14:39:34.011174 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:34.011256 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 14:39:34.011560 8652 catalog_manager.cc:4584] TS 2e6d064e084c4de885c2854edbd74e17 (127.21.43.65:38137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d7c46a14fc641119c41d3f7794339c4: Network error: Client connection negotiation failed: client connection to 127.21.43.65:38137: connect: Connection refused (error 111)
I20260810 14:39:34.017184 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9034
I20260810 14:39:34.024643 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 8634
2026-08-10T14:39:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1007 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:34 dist-test-slave-chsk krb5kdc[9225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:34 dist-test-slave-chsk krb5kdc[9225](info): set up 2 sockets
Aug 10 14:39:34 dist-test-slave-chsk krb5kdc[9225](info): commencing operation
krb5kdc: starting...
W20260810 14:39:35.641752 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.593s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:35 dist-test-slave-chsk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:39:35.783321 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39053
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:35.902684 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:35.902977 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:35.903064 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:35.907284 9241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:35.907359 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:35.907379 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:35.907397 9241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:35.907413 9241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:35.912978 9241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:35.914278 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:35.915400 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:35.924556 9248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:35.924772 9241 server_base.cc:1034] running on GCE node
W20260810 14:39:35.924583 9250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:35.924588 9247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:35.925526 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:35.926712 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:35.927901 9241 hybrid_clock.cc:648] HybridClock initialized: now 1786372775927882 us; error 34 us; skew 500 ppm
Aug 10 14:39:35 dist-test-slave-chsk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372775, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:35.937384 9241 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:39:35.938526 9241 webserver.cc:460] Webserver started at http://127.21.43.126:34123/ using document root <none> and password file <none>
I20260810 14:39:35.939224 9241 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:35.939322 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:35.939575 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:35.941838 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "fa624a655fd744e1abf84efc523b2cdc"
format_stamp: "Formatted at 2026-08-10 14:39:35 on dist-test-slave-chsk"
server_key: "8d5567594d4ab0a3ddf5fed9200828b9"
server_key_iv: "a6dc56c02e842d4a2745cb72dc1c3738"
server_key_version: "encryptionkey@0"
I20260810 14:39:35.942493 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "fa624a655fd744e1abf84efc523b2cdc"
format_stamp: "Formatted at 2026-08-10 14:39:35 on dist-test-slave-chsk"
server_key: "8d5567594d4ab0a3ddf5fed9200828b9"
server_key_iv: "a6dc56c02e842d4a2745cb72dc1c3738"
server_key_version: "encryptionkey@0"
I20260810 14:39:35.946722 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:35.949560 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:35.950845 9241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:35.951025 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "fa624a655fd744e1abf84efc523b2cdc"
format_stamp: "Formatted at 2026-08-10 14:39:35 on dist-test-slave-chsk"
server_key: "8d5567594d4ab0a3ddf5fed9200828b9"
server_key_iv: "a6dc56c02e842d4a2745cb72dc1c3738"
server_key_version: "encryptionkey@0"
I20260810 14:39:35.951195 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:35.982589 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:35.986095 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:35.986346 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:35.994012 9241 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:39053
I20260810 14:39:35.994040 9320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:39053 every 8 connection(s)
I20260810 14:39:35.995072 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:35.997984 9321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:36.002404 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9241
I20260810 14:39:36.002580 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:36.002907 21677 external_mini_cluster.cc:1428] Setting key a77f4d7367609a89f7dfd4f30a220293
I20260810 14:39:36.004844 9321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: Bootstrap starting.
I20260810 14:39:36.007308 9321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:36.008109 9321 log.cc:824] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:36.010442 9321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: No bootstrap required, opened a new log
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:36.014693 9321 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } }
I20260810 14:39:36.014991 9321 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:36.015074 9321 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa624a655fd744e1abf84efc523b2cdc, State: Initialized, Role: FOLLOWER
I20260810 14:39:36.015628 9321 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } }
I20260810 14:39:36.015805 9321 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:36.015908 9321 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:36.016026 9321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:36.017419 9321 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } }
I20260810 14:39:36.017846 9321 leader_election.cc:302] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa624a655fd744e1abf84efc523b2cdc; no voters:
I20260810 14:39:36.018227 9321 leader_election.cc:288] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:36.018419 9327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:36.018676 9327 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [term 1 LEADER]: Becoming Leader. State: Replica: fa624a655fd744e1abf84efc523b2cdc, State: Running, Role: LEADER
I20260810 14:39:36.018994 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.004526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.21.43.126:39053)
0810 14:39:36.005095 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.005113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.005151 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:39:36.005818 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.005835 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.006178 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.006526 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:36.007439 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.008548 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.009732 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.010223 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.010345 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.013308 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:36.013313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:36.013328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:36.013361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:36.015396 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.015901 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.015908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.015915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.016084 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.016457 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.016460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.016462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.017049 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:36.017298 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.017517 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.017867 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":151,"threads_started":1}
I20260810 14:39:36.019073 9327 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } }
I20260810 14:39:36.019712 9321 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:36.020358 9327 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa624a655fd744e1abf84efc523b2cdc. Latest consensus state: current_term: 1 leader_uuid: "fa624a655fd744e1abf84efc523b2cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } } }
I20260810 14:39:36.020485 9327 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:36.021214 9329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa624a655fd744e1abf84efc523b2cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa624a655fd744e1abf84efc523b2cdc" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39053 } } }
I20260810 14:39:36.021346 9329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:36.021827 9334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:36.025244 9334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:36.031256 9334 catalog_manager.cc:1347] Generated new cluster ID: c15cd5bb56d344808cbee9287672515b
I20260810 14:39:36.031344 9334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:36.045022 9334 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:36.046079 9334 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:36.053105 9334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: Generated new TSK 0
I20260810 14:39:36.053860 9334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:39:36.094113 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:36.213790 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:36.214124 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:36.214171 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:36.218410 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:36.218504 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:36.218567 9351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:36.224507 9351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:36.225672 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:36.226704 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:36.236948 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.236948 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.237037 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:36.237213 9351 server_base.cc:1034] running on GCE node
I20260810 14:39:36.237712 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:36.238596 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:36.239801 9351 hybrid_clock.cc:648] HybridClock initialized: now 1786372776239784 us; error 32 us; skew 500 ppm
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:36.242970 9351 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:39:36.244053 9351 webserver.cc:460] Webserver started at http://127.21.43.65:35315/ using document root <none> and password file <none>
I20260810 14:39:36.244680 9351 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:36.244750 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:36.245007 9351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:36.247350 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "547d8d02e9da41fb9a058a12af24ae76"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "e60b109179ca4899111b65239e035c1c"
server_key_iv: "3edd08de2da9e39d646aef413d47521e"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.247995 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "547d8d02e9da41fb9a058a12af24ae76"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "e60b109179ca4899111b65239e035c1c"
server_key_iv: "3edd08de2da9e39d646aef413d47521e"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.252533 9351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:39:36.255445 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.256805 9351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:36.256960 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "547d8d02e9da41fb9a058a12af24ae76"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "e60b109179ca4899111b65239e035c1c"
server_key_iv: "3edd08de2da9e39d646aef413d47521e"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.257089 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:36.275265 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:36.278692 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:36.278923 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:36.279672 9351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:36.280589 9351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:36.280643 9351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.280717 9351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:36.280750 9351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.290062 9351 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:44825
I20260810 14:39:36.290091 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:44825 every 8 connection(s)
I20260810 14:39:36.291059 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:36.292445 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9351
I20260810 14:39:36.292603 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:36.292907 21677 external_mini_cluster.cc:1428] Setting key cc213abb53e062b33b314f09b4297636
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:39:36.306684 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.293275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:43931 (local address 127.21.43.126:39053)
0810 14:39:36.293417 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.293420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.294437 (+ 1017us) server_negotiation.cc:407] Connection header received
0810 14:39:36.295989 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.295993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.296053 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.296163 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:36.297422 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.297977 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.299200 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.299439 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.299557 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.302548 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:36.302551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:36.302553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:36.302581 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:36.304242 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.304925 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.304928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.304930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.305012 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.305389 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.305393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.305394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.305643 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:36.305770 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.306354 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.306504 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:39:36.307651 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.293768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39053 (local address 127.21.43.65:43931)
0810 14:39:36.294277 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.294336 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.295766 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.296356 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.296363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.296749 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:36.297267 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.297281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.298113 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.298117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.299024 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.299034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.299623 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.299627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.299808 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.300508 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:36.300526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:36.302344 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:36.304392 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.304399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.304419 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.304800 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.305109 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.305112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.305114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.305278 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.305772 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:36.305781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:36.306069 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.306310 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.306672 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":139,"threads_started":1}
I20260810 14:39:36.309072 9498 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39053
I20260810 14:39:36.309363 9498 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:36.309918 9498 heartbeater.cc:511] Master 127.21.43.126:39053 requested a full tablet report, sending...
I20260810 14:39:36.311838 9277 ts_manager.cc:194] Registered new tserver with Master: 547d8d02e9da41fb9a058a12af24ae76 (127.21.43.65:44825)
I20260810 14:39:36.313684 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:43931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:39:36.326532 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:36.449261 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:36.449574 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:36.449621 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:36.453966 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:36.454051 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:36.454114 9503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:36.459975 9503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:36.461225 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:36.462368 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:36.472596 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.472628 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.472596 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:36.472846 9503 server_base.cc:1034] running on GCE node
I20260810 14:39:36.473433 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:36.474152 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:36.475312 9503 hybrid_clock.cc:648] HybridClock initialized: now 1786372776475293 us; error 34 us; skew 500 ppm
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:36.478180 9503 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:39:36.479115 9503 webserver.cc:460] Webserver started at http://127.21.43.66:46401/ using document root <none> and password file <none>
I20260810 14:39:36.479724 9503 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:36.479787 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:36.479960 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:36.482115 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "f648ffe8bf0b4caf82b2bbd868bc3722"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "98681705dc887cecc693874a02e5f257"
server_key_iv: "9b4d54220c890fa8f7f63a014df9071b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.482652 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "f648ffe8bf0b4caf82b2bbd868bc3722"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "98681705dc887cecc693874a02e5f257"
server_key_iv: "9b4d54220c890fa8f7f63a014df9071b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.486701 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:36.489503 9521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.490901 9503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:36.491077 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "f648ffe8bf0b4caf82b2bbd868bc3722"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "98681705dc887cecc693874a02e5f257"
server_key_iv: "9b4d54220c890fa8f7f63a014df9071b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.491267 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:36.520124 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:36.523910 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:36.524142 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:36.524822 9503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:36.525767 9503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:36.525818 9503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.525887 9503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:36.525923 9503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.535599 9503 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:44761
I20260810 14:39:36.535662 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:44761 every 8 connection(s)
I20260810 14:39:36.536593 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:36.545619 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9503
I20260810 14:39:36.545773 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:36.546159 21677 external_mini_cluster.cc:1428] Setting key b2423d2ff6a256c6ecb9ad6028cfd87d
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:36.551558 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.538541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:47607 (local address 127.21.43.126:39053)
0810 14:39:36.538687 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.538690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.539627 (+ 937us) server_negotiation.cc:407] Connection header received
0810 14:39:36.541011 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.541031 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.541097 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.541189 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:36.542416 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.542962 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.544178 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.544416 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.544530 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.547540 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:36.547544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:36.547547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:36.547578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:36.549094 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.549673 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.549680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.549684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.549794 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.550182 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.550188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.550192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.550465 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:36.550578 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.551121 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.551310 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:39:36.552371 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.538933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39053 (local address 127.21.43.66:47607)
0810 14:39:36.539469 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.539511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.540789 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.541370 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.541378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.541757 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:36.542267 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.542282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.543104 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.543107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.544052 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.544062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.544584 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.544588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.544753 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.545464 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:36.545490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:36.547311 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:36.549234 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.549241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.549258 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.549555 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.549916 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.549919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.549920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.550074 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.550579 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:36.550585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:36.550847 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.551007 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.551349 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":191,"threads_started":1}
I20260810 14:39:36.553767 9653 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39053
I20260810 14:39:36.554090 9653 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:36.554813 9653 heartbeater.cc:511] Master 127.21.43.126:39053 requested a full tablet report, sending...
I20260810 14:39:36.556244 9277 ts_manager.cc:194] Registered new tserver with Master: f648ffe8bf0b4caf82b2bbd868bc3722 (127.21.43.66:44761)
I20260810 14:39:36.556901 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:47607
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:39:36.579530 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:36.700901 9658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:36.701208 9658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:36.701254 9658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:36.705492 9658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:36.705585 9658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:36.705646 9658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:36.711364 9658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:36.712594 9658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:36.713707 9658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:36.723917 9666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.723944 9663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:36.723928 9664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:36.724205 9658 server_base.cc:1034] running on GCE node
I20260810 14:39:36.724722 9658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:36.725420 9658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:36.726630 9658 hybrid_clock.cc:648] HybridClock initialized: now 1786372776726616 us; error 32 us; skew 500 ppm
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:36.729779 9658 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:39:36.730731 9658 webserver.cc:460] Webserver started at http://127.21.43.67:33641/ using document root <none> and password file <none>
I20260810 14:39:36.731417 9658 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:36.731513 9658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:36.731746 9658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:36.733891 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "96a6cdd39ca94223b21fd45728cfdfa9"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "90f71fa87742bdb2dbc2e026d8ccdc40"
server_key_iv: "5750ebab5208db7f38e64cbf9523125b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.734498 9658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "96a6cdd39ca94223b21fd45728cfdfa9"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "90f71fa87742bdb2dbc2e026d8ccdc40"
server_key_iv: "5750ebab5208db7f38e64cbf9523125b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.738598 9658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:36.741266 9674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.742491 9658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:39:36.742647 9658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "96a6cdd39ca94223b21fd45728cfdfa9"
format_stamp: "Formatted at 2026-08-10 14:39:36 on dist-test-slave-chsk"
server_key: "90f71fa87742bdb2dbc2e026d8ccdc40"
server_key_iv: "5750ebab5208db7f38e64cbf9523125b"
server_key_version: "encryptionkey@0"
I20260810 14:39:36.742772 9658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:36.757859 9658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:36.761332 9658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:36.761546 9658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:36.762215 9658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:36.763123 9658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:36.763252 9658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.763326 9658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:36.763371 9658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:36.772572 9658 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:37971
I20260810 14:39:36.772594 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:37971 every 8 connection(s)
I20260810 14:39:36.773639 9658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:36.777967 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9658
I20260810 14:39:36.778149 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:36.778533 21677 external_mini_cluster.cc:1428] Setting key badd35825d689798f1e8ca0cf2e6f66a
Aug 10 14:39:36 dist-test-slave-chsk krb5kdc[9225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372776, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:36.788563 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.775699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:59219 (local address 127.21.43.126:39053)
0810 14:39:36.775886 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.775891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.776852 (+ 961us) server_negotiation.cc:407] Connection header received
0810 14:39:36.778234 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.778238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.778305 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.778421 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:36.780066 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.780681 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.781911 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.782162 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.782251 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.784864 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:36.784867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:36.784869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:36.784896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:36.786474 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.786997 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.786999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.787001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.787102 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.787478 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.787480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.787481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.787678 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:36.787758 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.788306 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.788428 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 14:39:36.789480 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.776159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39053 (local address 127.21.43.67:59219)
0810 14:39:36.776692 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.776738 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.777978 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.778660 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.778668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.779240 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:36.779915 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.779935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.780826 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.780830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.781777 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.781787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.782307 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.782310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.782474 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.783071 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:36.783088 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:36.784673 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:36.786590 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.786604 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.786618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.786908 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.787244 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:36.787247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:36.787248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:36.787395 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:36.787774 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:36.787780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:36.788048 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.788230 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.788508 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":128,"threads_started":1}
I20260810 14:39:36.790721 9809 heartbeater.cc:347] Connected to a master server at 127.21.43.126:39053
I20260810 14:39:36.790997 9809 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:36.791602 9809 heartbeater.cc:511] Master 127.21.43.126:39053 requested a full tablet report, sending...
I20260810 14:39:36.792675 9277 ts_manager.cc:194] Registered new tserver with Master: 96a6cdd39ca94223b21fd45728cfdfa9 (127.21.43.67:37971)
I20260810 14:39:36.793223 9277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:59219
I20260810 14:39:36.804586 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:36.814529 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.806714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.21.43.126:39053)
0810 14:39:36.806882 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.806888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.807069 (+ 181us) server_negotiation.cc:407] Connection header received
0810 14:39:36.807272 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.807274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.807324 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.807398 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:36.808129 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.808782 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.809594 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.809806 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.809886 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.811475 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:36.811478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:36.811480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:36.811506 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:36.812952 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.813403 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.813406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.813407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.813481 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:36.813737 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:36.813739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:36.813740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:36.813939 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:36.814048 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.814246 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.814325 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:39:36.817464 9277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:36.820684 9277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:36.835026 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.829480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:44825 (local address 127.0.0.1:43832)
0810 14:39:36.829967 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.830016 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.830161 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.830905 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.830910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.830930 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:36.831041 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.831046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.832441 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.832447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.833328 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.833337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.834464 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.834467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.834585 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.834633 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.834733 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.834866 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":99,"threads_started":1}
I20260810 14:39:36.835937 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.829639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.21.43.65:44825)
0810 14:39:36.830060 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.830070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.830096 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:39:36.830381 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.830388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.830619 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.830770 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:36.831191 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.832335 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.833464 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.834303 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.835460 (+ 1157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.835614 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.835670 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.835784 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":115,"threads_started":1}
I20260810 14:39:36.836165 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.830399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.21.43.67:37971)
0810 14:39:36.830781 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.830793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.831391 (+ 598us) server_negotiation.cc:407] Connection header received
0810 14:39:36.831627 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.831634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.831789 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.831878 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:36.832208 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.833399 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.834833 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.835569 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.835690 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.835827 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.835912 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.836000 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":151,"threads_started":1}
I20260810 14:39:36.836361 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.829647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44761 (local address 127.0.0.1:45530)
0810 14:39:36.831295 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.831310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.831406 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.832066 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.832069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.832086 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:36.832171 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.832176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.833515 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.833518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.834644 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.834652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.835941 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.835946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.836047 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.836080 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.836157 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.836237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1565,"thread_start_us":127,"threads_started":1}
I20260810 14:39:36.836407 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.830214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:37971 (local address 127.0.0.1:57634)
0810 14:39:36.831297 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:36.831310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:36.831460 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:36.832022 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:36.832024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:36.832038 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:36.832105 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.832109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.833586 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.833589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.834707 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:36.834716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.835692 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.835695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:36.835780 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.835803 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:39:36.835882 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:39:36.835954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":99,"threads_started":1}
I20260810 14:39:36.836948 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:36.830515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.21.43.66:44761)
0810 14:39:36.830974 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:36.830987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:36.831510 (+ 523us) server_negotiation.cc:407] Connection header received
0810 14:39:36.831594 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:36.831598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:36.831770 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:36.831911 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:36.832284 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.833398 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.834784 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:36.835642 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:36.835744 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:36.835870 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:36.836698 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 14:39:36.836801 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":162,"threads_started":1}
I20260810 14:39:36.839403 9580 tablet_service.cc:1467] Processing CreateTablet for tablet 7936db88de984e4dbc735d84565c77a3 (DEFAULT_TABLE table=test-table [id=9ac28cf44abf42bca0ce90ddc45014fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:36.839407 9425 tablet_service.cc:1467] Processing CreateTablet for tablet 7936db88de984e4dbc735d84565c77a3 (DEFAULT_TABLE table=test-table [id=9ac28cf44abf42bca0ce90ddc45014fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:36.839408 9730 tablet_service.cc:1467] Processing CreateTablet for tablet 7936db88de984e4dbc735d84565c77a3 (DEFAULT_TABLE table=test-table [id=9ac28cf44abf42bca0ce90ddc45014fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:36.840432 9580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7936db88de984e4dbc735d84565c77a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:36.840452 9425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7936db88de984e4dbc735d84565c77a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:36.840458 9730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7936db88de984e4dbc735d84565c77a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:36.847536 9828 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Bootstrap starting.
I20260810 14:39:36.848202 9830 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Bootstrap starting.
I20260810 14:39:36.849864 9828 tablet_bootstrap.cc:654] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:36.850066 9830 tablet_bootstrap.cc:654] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:36.851066 9828 log.cc:824] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:36.851078 9830 log.cc:824] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:36.851531 9829 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Bootstrap starting.
I20260810 14:39:36.853266 9830 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: No bootstrap required, opened a new log
I20260810 14:39:36.853332 9828 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: No bootstrap required, opened a new log
I20260810 14:39:36.853487 9830 ts_tablet_manager.cc:1397] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 14:39:36.853526 9828 ts_tablet_manager.cc:1397] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:39:36.853901 9829 tablet_bootstrap.cc:654] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:36.854810 9829 log.cc:824] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:36.857023 9829 tablet_bootstrap.cc:492] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: No bootstrap required, opened a new log
I20260810 14:39:36.857244 9829 ts_tablet_manager.cc:1397] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 14:39:36.857167 9828 raft_consensus.cc:348] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.857404 9828 raft_consensus.cc:374] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:36.857190 9830 raft_consensus.cc:348] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.857438 9828 raft_consensus.cc:729] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f648ffe8bf0b4caf82b2bbd868bc3722, State: Initialized, Role: FOLLOWER
I20260810 14:39:36.857460 9830 raft_consensus.cc:374] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:36.857522 9830 raft_consensus.cc:729] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a6cdd39ca94223b21fd45728cfdfa9, State: Initialized, Role: FOLLOWER
I20260810 14:39:36.857897 9828 consensus_queue.cc:260] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.857987 9830 consensus_queue.cc:260] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.858654 9653 heartbeater.cc:503] Master 127.21.43.126:39053 was elected leader, sending a full tablet report...
I20260810 14:39:36.858984 9828 ts_tablet_manager.cc:1428] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 14:39:36.859009 9830 ts_tablet_manager.cc:1428] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260810 14:39:36.859344 9809 heartbeater.cc:503] Master 127.21.43.126:39053 was elected leader, sending a full tablet report...
I20260810 14:39:36.862453 9829 raft_consensus.cc:348] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.862748 9829 raft_consensus.cc:374] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:36.862830 9829 raft_consensus.cc:729] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547d8d02e9da41fb9a058a12af24ae76, State: Initialized, Role: FOLLOWER
I20260810 14:39:36.863358 9829 consensus_queue.cc:260] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:36.864070 9498 heartbeater.cc:503] Master 127.21.43.126:39053 was elected leader, sending a full tablet report...
I20260810 14:39:36.864351 9829 ts_tablet_manager.cc:1428] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260810 14:39:37.024780 9810 tablet.cc:2369] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:39:37.037938 9654 tablet.cc:2369] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:39:37.042476 9499 tablet.cc:2369] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:39:37.228044 9838 raft_consensus.cc:482] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:37.228255 9838 raft_consensus.cc:504] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:37.229411 9838 leader_election.cc:288] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f648ffe8bf0b4caf82b2bbd868bc3722 (127.21.43.66:44761), 96a6cdd39ca94223b21fd45728cfdfa9 (127.21.43.67:37971)
I20260810 14:39:37.233292 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.229740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:45667 (local address 127.21.43.66:44761)
0810 14:39:37.229892 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:37.229898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:37.230116 (+ 218us) server_negotiation.cc:407] Connection header received
0810 14:39:37.230238 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:37.230242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:37.230303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:37.230424 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:37.230738 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.231293 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.232355 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.232748 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.232874 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.232927 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:37.233119 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 14:39:37.233183 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:39:37.233306 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.229661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:44761 (local address 127.21.43.65:45667)
0810 14:39:37.229973 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:37.229991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:37.230122 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:37.230441 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:37.230444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:37.230464 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:37.230610 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.230616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.231407 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.231411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.232227 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.232236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.232870 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.232874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.232969 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.232993 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:39:37.233086 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:39:37.233157 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
I20260810 14:39:37.233457 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.230262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:42117 (local address 127.21.43.67:37971)
0810 14:39:37.230417 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:37.230425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:37.230446 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:39:37.230605 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:37.230608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:37.230653 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:37.230733 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:37.230995 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.231490 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.232499 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.232881 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.233002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.233056 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:37.233245 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 14:39:37.233314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:39:37.233477 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.230088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:37971 (local address 127.21.43.65:42117)
0810 14:39:37.230387 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:37.230400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:37.230495 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:37.230746 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:37.230749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:37.230765 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:37.230886 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.230891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.231611 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.231615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.232369 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.232378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.232996 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.233000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.233124 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.233142 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:39:37.233227 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:39:37.233294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260810 14:39:37.234174 9753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7936db88de984e4dbc735d84565c77a3" candidate_uuid: "547d8d02e9da41fb9a058a12af24ae76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" is_pre_election: true
I20260810 14:39:37.234181 9601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7936db88de984e4dbc735d84565c77a3" candidate_uuid: "547d8d02e9da41fb9a058a12af24ae76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" is_pre_election: true
I20260810 14:39:37.234483 9753 raft_consensus.cc:2454] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547d8d02e9da41fb9a058a12af24ae76 in term 0.
I20260810 14:39:37.234493 9601 raft_consensus.cc:2454] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 547d8d02e9da41fb9a058a12af24ae76 in term 0.
I20260810 14:39:37.234957 9370 leader_election.cc:302] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547d8d02e9da41fb9a058a12af24ae76, 96a6cdd39ca94223b21fd45728cfdfa9; no voters:
I20260810 14:39:37.235219 9838 raft_consensus.cc:2781] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:37.235311 9838 raft_consensus.cc:482] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:37.235363 9838 raft_consensus.cc:3037] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:37.236398 9838 raft_consensus.cc:504] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:37.236718 9838 leader_election.cc:288] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [CANDIDATE]: Term 1 election: Requested vote from peers f648ffe8bf0b4caf82b2bbd868bc3722 (127.21.43.66:44761), 96a6cdd39ca94223b21fd45728cfdfa9 (127.21.43.67:37971)
I20260810 14:39:37.237071 9601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7936db88de984e4dbc735d84565c77a3" candidate_uuid: "547d8d02e9da41fb9a058a12af24ae76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722"
I20260810 14:39:37.237181 9753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7936db88de984e4dbc735d84565c77a3" candidate_uuid: "547d8d02e9da41fb9a058a12af24ae76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a6cdd39ca94223b21fd45728cfdfa9"
I20260810 14:39:37.237197 9601 raft_consensus.cc:3037] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:37.237329 9753 raft_consensus.cc:3037] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:37.238245 9601 raft_consensus.cc:2454] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547d8d02e9da41fb9a058a12af24ae76 in term 1.
I20260810 14:39:37.238437 9753 raft_consensus.cc:2454] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 547d8d02e9da41fb9a058a12af24ae76 in term 1.
I20260810 14:39:37.238562 9370 leader_election.cc:302] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547d8d02e9da41fb9a058a12af24ae76, f648ffe8bf0b4caf82b2bbd868bc3722; no voters:
I20260810 14:39:37.238770 9838 raft_consensus.cc:2781] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:37.238983 9838 raft_consensus.cc:686] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 1 LEADER]: Becoming Leader. State: Replica: 547d8d02e9da41fb9a058a12af24ae76, State: Running, Role: LEADER
I20260810 14:39:37.239354 9838 consensus_queue.cc:237] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } }
I20260810 14:39:37.242491 9277 catalog_manager.cc:5495] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 547d8d02e9da41fb9a058a12af24ae76 (127.21.43.65). New cstate: current_term: 1 leader_uuid: "547d8d02e9da41fb9a058a12af24ae76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "547d8d02e9da41fb9a058a12af24ae76" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 } health_report { overall_health: UNKNOWN } } }
I20260810 14:39:37.289652 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.286145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.21.43.65:44825)
0810 14:39:37.286303 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:37.286310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:37.286325 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:39:37.286397 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:37.286400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:37.286452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:37.286560 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:37.286807 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.287341 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.288185 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.288393 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.288501 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.288820 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:37.289202 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:37.289363 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:37.289471 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 14:39:37.289511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:39:37.296051 9753 raft_consensus.cc:1261] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 1 FOLLOWER]: Refusing update from remote peer 547d8d02e9da41fb9a058a12af24ae76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:37.296063 9601 raft_consensus.cc:1261] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 1 FOLLOWER]: Refusing update from remote peer 547d8d02e9da41fb9a058a12af24ae76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:37.296763 9838 consensus_queue.cc:1035] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f648ffe8bf0b4caf82b2bbd868bc3722" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 44761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:37.297384 9842 consensus_queue.cc:1035] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a6cdd39ca94223b21fd45728cfdfa9" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:37.305949 9846 mvcc.cc:205] Tried to move back new op lower bound from 7316982895796436992 to 7316982895575171072. Current Snapshot: MvccSnapshot[applied={T|T < 7316982895796436992}]
I20260810 14:39:37.306147 9849 mvcc.cc:205] Tried to move back new op lower bound from 7316982895796436992 to 7316982895575171072. Current Snapshot: MvccSnapshot[applied={T|T < 7316982895796436992}]
I20260810 14:39:37.313524 9279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:37.313680 9279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:37.316179 9279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa624a655fd744e1abf84efc523b2cdc: Sending DeleteTablet for 3 replicas of tablet 7936db88de984e4dbc735d84565c77a3
I20260810 14:39:37.316998 9425 tablet_service.cc:1514] Processing DeleteTablet for tablet 7936db88de984e4dbc735d84565c77a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:37 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:43832
I20260810 14:39:37.317116 9580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7936db88de984e4dbc735d84565c77a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:37 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260810 14:39:37.317224 9730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7936db88de984e4dbc735d84565c77a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:37 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57634
I20260810 14:39:37.317507 9858 tablet_replica.cc:330] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76: stopping tablet replica
I20260810 14:39:37.317832 9858 raft_consensus.cc:2229] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:39:37.318082 9860 tablet_replica.cc:330] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: stopping tablet replica
I20260810 14:39:37.318233 9858 raft_consensus.cc:2258] T 7936db88de984e4dbc735d84565c77a3 P 547d8d02e9da41fb9a058a12af24ae76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:37.318531 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9351
I20260810 14:39:37.321709 9860 raft_consensus.cc:2229] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:37.322111 9859 tablet_replica.cc:330] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: stopping tablet replica
I20260810 14:39:37.322130 9860 raft_consensus.cc:2258] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:37.322435 9859 raft_consensus.cc:2229] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:37.322770 9859 raft_consensus.cc:2258] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:37.323994 9859 ts_tablet_manager.cc:1905] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:37.323979 9860 ts_tablet_manager.cc:1905] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:37.326200 9859 ts_tablet_manager.cc:1918] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:37.326313 9859 log.cc:1196] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/7936db88de984e4dbc735d84565c77a3
I20260810 14:39:37.326602 9859 ts_tablet_manager.cc:1939] T 7936db88de984e4dbc735d84565c77a3 P f648ffe8bf0b4caf82b2bbd868bc3722: Deleting consensus metadata
I20260810 14:39:37.327469 9860 ts_tablet_manager.cc:1918] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:37.327586 9860 log.cc:1196] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/7936db88de984e4dbc735d84565c77a3
I20260810 14:39:37.327903 9860 ts_tablet_manager.cc:1939] T 7936db88de984e4dbc735d84565c77a3 P 96a6cdd39ca94223b21fd45728cfdfa9: Deleting consensus metadata
I20260810 14:39:37.327916 9261 catalog_manager.cc:4841] TS f648ffe8bf0b4caf82b2bbd868bc3722 (127.21.43.66:44761): tablet 7936db88de984e4dbc735d84565c77a3 (table test-table [id=9ac28cf44abf42bca0ce90ddc45014fe]) successfully deleted
I20260810 14:39:37.329227 9261 catalog_manager.cc:4841] TS 96a6cdd39ca94223b21fd45728cfdfa9 (127.21.43.67:37971): tablet 7936db88de984e4dbc735d84565c77a3 (table test-table [id=9ac28cf44abf42bca0ce90ddc45014fe]) successfully deleted
W20260810 14:39:37.329749 9262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:44825 (error 108)
I20260810 14:39:37.330070 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9503
I20260810 14:39:37.332706 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.332434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:44825 (local address 127.0.0.1:43862)
0810 14:39:37.332565 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:37.332636 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:44825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 14:39:37.332854 9262 catalog_manager.cc:4584] TS 547d8d02e9da41fb9a058a12af24ae76 (127.21.43.65:44825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7936db88de984e4dbc735d84565c77a3: Network error: Client connection negotiation failed: client connection to 127.21.43.65:44825: connect: Connection refused (error 111)
I20260810 14:39:37.337613 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9658
I20260810 14:39:37.344636 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9241
2026-08-10T14:39:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3319 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T14:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:37Z Disabled control of system clock
I20260810 14:39:37.377861 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43295
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:37.487294 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:37.487552 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:37.487587 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:37.491473 9867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:37.491523 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:37.491539 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:37.491556 9867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:37.491571 9867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:37.496225 9867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:37.497217 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:37.498541 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:37.506901 9873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:37.506863 9874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:37.506948 9876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:37.507069 9867 server_base.cc:1034] running on GCE node
I20260810 14:39:37.507664 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:37.508752 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:37.509960 9867 hybrid_clock.cc:648] HybridClock initialized: now 1786372777509946 us; error 25 us; skew 500 ppm
I20260810 14:39:37.511860 9867 webserver.cc:460] Webserver started at http://127.21.43.126:42269/ using document root <none> and password file <none>
I20260810 14:39:37.512439 9867 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:37.512533 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:37.512777 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:37.514819 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "0c7af0c44357443495e60b671612405b"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "7cf430a933724b4cc1ffefae61d1707f"
server_key_iv: "d0077e26abd985d858c66964fdb8baa1"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.515417 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "0c7af0c44357443495e60b671612405b"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "7cf430a933724b4cc1ffefae61d1707f"
server_key_iv: "d0077e26abd985d858c66964fdb8baa1"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.518982 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:37.521400 9885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.522537 9867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:37.522679 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "0c7af0c44357443495e60b671612405b"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "7cf430a933724b4cc1ffefae61d1707f"
server_key_iv: "d0077e26abd985d858c66964fdb8baa1"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.522791 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:37.534579 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:37.535362 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:37.535552 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:37.542654 9867 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43295
I20260810 14:39:37.542686 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43295 every 8 connection(s)
I20260810 14:39:37.543628 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:37.546484 9944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:37.552117 9944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: Bootstrap starting.
I20260810 14:39:37.553619 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9867
I20260810 14:39:37.553776 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:37.554047 21677 external_mini_cluster.cc:1428] Setting key 56de1a8319586166ebd5c5844bfb5a55
I20260810 14:39:37.554822 9944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:37.555737 9944 log.cc:824] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:37.557821 9944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: No bootstrap required, opened a new log
I20260810 14:39:37.562156 9944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } }
I20260810 14:39:37.562371 9944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:37.562439 9944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7af0c44357443495e60b671612405b, State: Initialized, Role: FOLLOWER
I20260810 14:39:37.562876 9944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } }
I20260810 14:39:37.562999 9944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:37.563081 9944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:37.563089 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.555232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.21.43.126:43295)
0810 14:39:37.555899 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:37.555916 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:37.555952 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:39:37.556587 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:37.556603 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:37.556915 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:37.557198 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:37.557531 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.558530 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.559768 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.560239 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.560353 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.561246 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:37.561251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:37.561262 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:37.561633 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:37.561687 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:37.561706 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:37.561782 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:37.561942 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 14:39:37.562234 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260810 14:39:37.563232 9944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:37.564102 9944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } }
I20260810 14:39:37.564435 9944 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7af0c44357443495e60b671612405b; no voters:
I20260810 14:39:37.564742 9944 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:37.564819 9951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:37.565004 9951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [term 1 LEADER]: Becoming Leader. State: Replica: 0c7af0c44357443495e60b671612405b, State: Running, Role: LEADER
I20260810 14:39:37.565289 9951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } }
I20260810 14:39:37.566010 9944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:37.567121 9953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7af0c44357443495e60b671612405b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } } }
I20260810 14:39:37.567284 9953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:37.567582 9952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7af0c44357443495e60b671612405b. Latest consensus state: current_term: 1 leader_uuid: "0c7af0c44357443495e60b671612405b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7af0c44357443495e60b671612405b" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43295 } } }
I20260810 14:39:37.567690 9960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:37.567715 9952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:37.570919 9960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:37.576995 9960 catalog_manager.cc:1347] Generated new cluster ID: c5f7dd7ed2954dd9b75c2d19159bbf6d
I20260810 14:39:37.577085 9960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:37.582129 9960 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:37.583097 9960 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:37.593642 9960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: Generated new TSK 0
I20260810 14:39:37.594368 9960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:37.602612 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--builtin_ntp_servers=127.21.43.84:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:37.718603 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:37.718928 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:37.719008 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:37.723285 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:37.723372 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:37.723466 9970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:37.729022 9970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:37.731070 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:37.733261 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:37.746959 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:37.747201 9970 server_base.cc:1034] running on GCE node
W20260810 14:39:37.746959 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:37.746971 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:37.747712 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:37.748395 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:37.749553 9970 hybrid_clock.cc:648] HybridClock initialized: now 1786372777749521 us; error 50 us; skew 500 ppm
I20260810 14:39:37.751492 9970 webserver.cc:460] Webserver started at http://127.21.43.65:43407/ using document root <none> and password file <none>
I20260810 14:39:37.752104 9970 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:37.752172 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:37.752409 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:37.754510 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "08238d548c6b40be8f9c4838b6eaed32"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "d4327d417e0f27ef5ad43d67108d5a74"
server_key_iv: "689fc789fead05237648529b8adf7ccd"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.755110 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "08238d548c6b40be8f9c4838b6eaed32"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "d4327d417e0f27ef5ad43d67108d5a74"
server_key_iv: "689fc789fead05237648529b8adf7ccd"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.759044 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:39:37.761713 9984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.763080 9970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:39:37.763255 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "08238d548c6b40be8f9c4838b6eaed32"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "d4327d417e0f27ef5ad43d67108d5a74"
server_key_iv: "689fc789fead05237648529b8adf7ccd"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.763373 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:37.780189 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:37.780946 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:37.781153 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:37.781826 9970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:37.782840 9970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:37.782889 9970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.782958 9970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:37.782990 9970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.792358 9970 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:34119
I20260810 14:39:37.792384 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:34119 every 8 connection(s)
I20260810 14:39:37.793329 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:37.800828 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 9970
I20260810 14:39:37.800993 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:37.801306 21677 external_mini_cluster.cc:1428] Setting key fe18576b54250dc570fe174d3aa7705e
I20260810 14:39:37.802866 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.795256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:47951 (local address 127.21.43.126:43295)
0810 14:39:37.795404 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:37.795407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:37.796491 (+ 1084us) server_negotiation.cc:407] Connection header received
0810 14:39:37.797942 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:37.797945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:37.798012 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:37.798110 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:37.799030 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.799648 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.800783 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.801073 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.801177 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.802370 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:37.802374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:37.802377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:37.802390 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:37.802418 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:37.802427 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:37.802500 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:37.802660 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 14:39:37.802725 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:39:37.803460 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--builtin_ntp_servers=127.21.43.84:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:37.803886 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:37.795690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43295 (local address 127.21.43.65:47951)
0810 14:39:37.796345 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:37.796386 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:37.797718 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:37.798299 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:37.798305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:37.798390 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:37.798886 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.798898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.799786 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.799791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.800655 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:37.800664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:37.801227 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:37.801231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:37.801419 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:37.802096 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:37.802097 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:37.802144 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:37.802210 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:37.802514 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:37.802529 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:37.802535 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:39:37.802657 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 14:39:37.802961 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260810 14:39:37.805258 10118 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43295
I20260810 14:39:37.805514 10118 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:37.806075 10118 heartbeater.cc:511] Master 127.21.43.126:43295 requested a full tablet report, sending...
I20260810 14:39:37.807857 9904 ts_manager.cc:194] Registered new tserver with Master: 08238d548c6b40be8f9c4838b6eaed32 (127.21.43.65:34119)
I20260810 14:39:37.809675 9904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.65:47951
W20260810 14:39:37.926421 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:37.926733 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:37.926776 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:37.930998 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:37.931077 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:37.931187 10121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:37.936331 10121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:37.937500 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:37.938913 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:37.949751 10129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:37.949726 10126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:37.949716 10127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:37.950222 10121 server_base.cc:1034] running on GCE node
I20260810 14:39:37.950691 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:37.951448 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:37.952664 10121 hybrid_clock.cc:648] HybridClock initialized: now 1786372777952640 us; error 33 us; skew 500 ppm
I20260810 14:39:37.954627 10121 webserver.cc:460] Webserver started at http://127.21.43.66:33489/ using document root <none> and password file <none>
I20260810 14:39:37.955322 10121 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:37.955426 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:37.955682 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:37.958308 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "eb6f4341b7bf46b8964b7d438b028e38"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "9ab1093e5c346134c59c4944130de0ee"
server_key_iv: "805ddfd63426170543ee5932b1a96c5b"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.958986 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6f4341b7bf46b8964b7d438b028e38"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "9ab1093e5c346134c59c4944130de0ee"
server_key_iv: "805ddfd63426170543ee5932b1a96c5b"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.963599 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:39:37.966468 10136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.968225 10121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:37.968406 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "eb6f4341b7bf46b8964b7d438b028e38"
format_stamp: "Formatted at 2026-08-10 14:39:37 on dist-test-slave-chsk"
server_key: "9ab1093e5c346134c59c4944130de0ee"
server_key_iv: "805ddfd63426170543ee5932b1a96c5b"
server_key_version: "encryptionkey@0"
I20260810 14:39:37.968542 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:37.990537 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:37.991400 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:37.991619 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:37.992273 10121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:37.993301 10121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:37.993374 10121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:37.993446 10121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:37.993492 10121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:38.002544 10121 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:35847
I20260810 14:39:38.002569 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:35847 every 8 connection(s)
I20260810 14:39:38.003676 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:38.013164 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10121
I20260810 14:39:38.013352 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:38.013355 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.005751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:34419 (local address 127.21.43.126:43295)
0810 14:39:38.005913 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.005917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.007008 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 14:39:38.008532 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.008536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.008596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.008704 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:38.009626 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.010167 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.011390 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.011614 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.011727 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.012849 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:38.012852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:38.012854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:38.012866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:38.012884 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:38.012892 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:38.012965 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.013133 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.013200 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:39:38.013707 21677 external_mini_cluster.cc:1428] Setting key b09b2314761e4b1eefb6636e3927cac4
I20260810 14:39:38.014390 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.006154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43295 (local address 127.21.43.66:34419)
0810 14:39:38.006853 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.006897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.008308 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.008889 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.008896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.008981 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:38.009469 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.009482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.010302 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.010308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.011265 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.011274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.011777 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.011781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.011953 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.012574 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:38.012576 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:38.012615 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:38.012679 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:38.012980 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:38.012992 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:38.013003 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.013133 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.013426 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260810 14:39:38.015623 10273 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43295
I20260810 14:39:38.015873 10273 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:38.015903 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--builtin_ntp_servers=127.21.43.84:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:38.016444 10273 heartbeater.cc:511] Master 127.21.43.126:43295 requested a full tablet report, sending...
I20260810 14:39:38.017524 9902 ts_manager.cc:194] Registered new tserver with Master: eb6f4341b7bf46b8964b7d438b028e38 (127.21.43.66:35847)
I20260810 14:39:38.018131 9902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.66:34419
W20260810 14:39:38.134617 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:38.134944 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:38.135020 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:38.139480 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:38.139571 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:38.139662 10276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:38.144719 10276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:38.145900 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:38.147359 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:38.157277 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:38.157281 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:38.157277 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:38.157724 10276 server_base.cc:1034] running on GCE node
I20260810 14:39:38.158234 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:38.159107 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:38.160346 10276 hybrid_clock.cc:648] HybridClock initialized: now 1786372778160322 us; error 35 us; skew 500 ppm
I20260810 14:39:38.162415 10276 webserver.cc:460] Webserver started at http://127.21.43.67:44365/ using document root <none> and password file <none>
I20260810 14:39:38.163066 10276 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:38.163195 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:38.163440 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:38.165719 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "536925b43c88439b89c2d85393842b91"
format_stamp: "Formatted at 2026-08-10 14:39:38 on dist-test-slave-chsk"
server_key: "aab15e76d4a44ec0e6e458e0ed53fa08"
server_key_iv: "4a2410ce5a265891dd4965c5dff67fe6"
server_key_version: "encryptionkey@0"
I20260810 14:39:38.166338 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "536925b43c88439b89c2d85393842b91"
format_stamp: "Formatted at 2026-08-10 14:39:38 on dist-test-slave-chsk"
server_key: "aab15e76d4a44ec0e6e458e0ed53fa08"
server_key_iv: "4a2410ce5a265891dd4965c5dff67fe6"
server_key_version: "encryptionkey@0"
I20260810 14:39:38.170897 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 14:39:38.173785 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:38.175258 10276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:38.175412 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "536925b43c88439b89c2d85393842b91"
format_stamp: "Formatted at 2026-08-10 14:39:38 on dist-test-slave-chsk"
server_key: "aab15e76d4a44ec0e6e458e0ed53fa08"
server_key_iv: "4a2410ce5a265891dd4965c5dff67fe6"
server_key_version: "encryptionkey@0"
I20260810 14:39:38.175554 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:38.197132 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:38.197896 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:38.198123 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:38.198769 10276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:38.199844 10276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:38.199920 10276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:38.199996 10276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:38.200037 10276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:38.209259 10276 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:45039
I20260810 14:39:38.209277 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:45039 every 8 connection(s)
I20260810 14:39:38.210260 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:38.214365 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10276
I20260810 14:39:38.214478 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:38.214774 21677 external_mini_cluster.cc:1428] Setting key 809b745cfe8e64eaccce72cac779d022
I20260810 14:39:38.220221 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.212266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:33017 (local address 127.21.43.126:43295)
0810 14:39:38.212427 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.212432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.213555 (+ 1123us) server_negotiation.cc:407] Connection header received
0810 14:39:38.215071 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.215075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.215200 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.215286 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:38.216544 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.217219 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.218359 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.218576 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.218655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.219807 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:38.219810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:38.219811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:38.219823 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:38.219842 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:38.219849 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:38.219907 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.220048 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.220101 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:39:38.221108 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.212711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43295 (local address 127.21.43.67:33017)
0810 14:39:38.213386 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.213430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.214803 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.215507 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.215514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.215636 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:38.216388 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.216410 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.217352 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.217355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.218253 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.218262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.218715 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.218717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.218857 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.219563 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:38.219564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:38.219606 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:38.219674 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:38.219925 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:38.219931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:38.219936 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.220032 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.220280 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260810 14:39:38.222242 10418 heartbeater.cc:347] Connected to a master server at 127.21.43.126:43295
I20260810 14:39:38.222491 10418 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:38.223034 10418 heartbeater.cc:511] Master 127.21.43.126:43295 requested a full tablet report, sending...
I20260810 14:39:38.224195 9902 ts_manager.cc:194] Registered new tserver with Master: 536925b43c88439b89c2d85393842b91 (127.21.43.67:45039)
I20260810 14:39:38.224763 9902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.67:33017
I20260810 14:39:38.228919 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:38.235240 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.230961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.21.43.126:43295)
0810 14:39:38.231180 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.231185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.231332 (+ 147us) server_negotiation.cc:407] Connection header received
0810 14:39:38.231486 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.231490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.231544 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.231633 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:38.231960 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.232495 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.233361 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.233589 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.233700 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.234657 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:38.234661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:38.234664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:38.234677 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:38.234707 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:38.234715 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:38.234854 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.234986 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.235029 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:39:38.238551 9902 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:38.241755 9902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:38.257279 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.251794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.21.43.67:45039)
0810 14:39:38.252188 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.252202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.252431 (+ 229us) server_negotiation.cc:407] Connection header received
0810 14:39:38.252669 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.252676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.252852 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.252973 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.253487 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254462 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256009 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.256587 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256728 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.256943 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.257004 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.257099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":164,"threads_started":1}
I20260810 14:39:38.257274 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.251191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45039 (local address 127.0.0.1:57046)
0810 14:39:38.252327 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.252342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.252506 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.253182 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.253186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.253204 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.253356 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.253361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254586 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.254589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.255870 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.255880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.256729 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.256838 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.256876 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.256981 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.257100 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":227,"thread_start_us":114,"threads_started":1}
I20260810 14:39:38.257690 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.251792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35847 (local address 127.0.0.1:42988)
0810 14:39:38.252475 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.252487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.252590 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.253197 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.253200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.253212 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.253351 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.253359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254478 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.254481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.255974 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.255983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.257147 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.257151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.257322 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.257369 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.257443 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.257541 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":119,"spinlock_wait_cycles":1408,"thread_start_us":111,"threads_started":1}
I20260810 14:39:38.257916 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.251098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.21.43.65:34119)
0810 14:39:38.251618 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.251632 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.252122 (+ 490us) server_negotiation.cc:407] Connection header received
0810 14:39:38.252211 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.252217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.252435 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.252598 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.253007 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254001 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.255960 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.256761 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256888 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.257075 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.257525 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.257624 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":152,"threads_started":1}
I20260810 14:39:38.258162 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.251425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.21.43.66:35847)
0810 14:39:38.252341 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.252354 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.252691 (+ 337us) server_negotiation.cc:407] Connection header received
0810 14:39:38.252767 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.252771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.252923 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.253053 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.253517 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254350 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256072 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.256817 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.256909 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.257076 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.257883 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.257999 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":155,"threads_started":1}
I20260810 14:39:38.258507 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.250933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34119 (local address 127.0.0.1:60202)
0810 14:39:38.251766 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.251821 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.251964 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.252754 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.252759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.252780 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.252884 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.252889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.254131 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.254135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.255822 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.255834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.257147 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.257151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.257325 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.257372 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.257443 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.258367 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":1058,"thread_start_us":92,"threads_started":1}
I20260810 14:39:38.260604 10343 tablet_service.cc:1467] Processing CreateTablet for tablet 606442b7a2b74aac9bb679f5599fc65b (DEFAULT_TABLE table=test-table [id=c1bc68e77c2b40889269e26763d46dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:38.260605 10194 tablet_service.cc:1467] Processing CreateTablet for tablet 606442b7a2b74aac9bb679f5599fc65b (DEFAULT_TABLE table=test-table [id=c1bc68e77c2b40889269e26763d46dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:38.261660 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606442b7a2b74aac9bb679f5599fc65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:38.261662 10343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606442b7a2b74aac9bb679f5599fc65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:38.266264 10043 tablet_service.cc:1467] Processing CreateTablet for tablet 606442b7a2b74aac9bb679f5599fc65b (DEFAULT_TABLE table=test-table [id=c1bc68e77c2b40889269e26763d46dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:38.267529 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606442b7a2b74aac9bb679f5599fc65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:38.268886 10435 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Bootstrap starting.
I20260810 14:39:38.269767 10436 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Bootstrap starting.
I20260810 14:39:38.270973 10435 tablet_bootstrap.cc:654] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:38.271754 10435 log.cc:824] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:38.272377 10436 tablet_bootstrap.cc:654] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:38.273437 10436 log.cc:824] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:38.273929 10435 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: No bootstrap required, opened a new log
I20260810 14:39:38.274161 10435 ts_tablet_manager.cc:1397] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:39:38.275533 10440 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Bootstrap starting.
I20260810 14:39:38.275753 10436 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: No bootstrap required, opened a new log
I20260810 14:39:38.276067 10436 ts_tablet_manager.cc:1397] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:39:38.277921 10440 tablet_bootstrap.cc:654] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:38.278479 10435 raft_consensus.cc:348] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.278761 10435 raft_consensus.cc:374] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:38.278864 10435 raft_consensus.cc:729] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536925b43c88439b89c2d85393842b91, State: Initialized, Role: FOLLOWER
I20260810 14:39:38.279515 10435 consensus_queue.cc:260] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.279763 10440 log.cc:824] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:38.279815 10436 raft_consensus.cc:348] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.280079 10436 raft_consensus.cc:374] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:38.280197 10436 raft_consensus.cc:729] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f4341b7bf46b8964b7d438b028e38, State: Initialized, Role: FOLLOWER
I20260810 14:39:38.280683 10436 consensus_queue.cc:260] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.280766 10435 ts_tablet_manager.cc:1428] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 14:39:38.280979 10418 heartbeater.cc:503] Master 127.21.43.126:43295 was elected leader, sending a full tablet report...
I20260810 14:39:38.281725 10436 ts_tablet_manager.cc:1428] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260810 14:39:38.282063 10440 tablet_bootstrap.cc:492] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: No bootstrap required, opened a new log
I20260810 14:39:38.282110 10273 heartbeater.cc:503] Master 127.21.43.126:43295 was elected leader, sending a full tablet report...
I20260810 14:39:38.282315 10440 ts_tablet_manager.cc:1397] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:39:38.286406 10440 raft_consensus.cc:348] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.286782 10440 raft_consensus.cc:374] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:38.286897 10440 raft_consensus.cc:729] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08238d548c6b40be8f9c4838b6eaed32, State: Initialized, Role: FOLLOWER
I20260810 14:39:38.287391 10440 consensus_queue.cc:260] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.288162 10118 heartbeater.cc:503] Master 127.21.43.126:43295 was elected leader, sending a full tablet report...
I20260810 14:39:38.288414 10440 ts_tablet_manager.cc:1428] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 14:39:38.294557 10119 tablet.cc:2369] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:39:38.364863 10441 raft_consensus.cc:482] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:38.365123 10441 raft_consensus.cc:504] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.366400 10441 leader_election.cc:288] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08238d548c6b40be8f9c4838b6eaed32 (127.21.43.65:34119), eb6f4341b7bf46b8964b7d438b028e38 (127.21.43.66:35847)
I20260810 14:39:38.370767 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.366829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:35063 (local address 127.21.43.66:35847)
0810 14:39:38.367006 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.367011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.367031 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:39:38.367246 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.367249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.367298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.367400 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.367761 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.368449 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.369620 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.370118 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.370224 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.370268 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.370527 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.370602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:39:38.370767 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.366699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:35847 (local address 127.21.43.67:35063)
0810 14:39:38.366902 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.366920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.367054 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.367441 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.367444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.367457 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.367629 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.367641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.368557 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.368561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.369490 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.369501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.370264 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.370268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.370378 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.370405 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.370496 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.370584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":2}
I20260810 14:39:38.370972 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.366699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34119 (local address 127.21.43.67:57431)
0810 14:39:38.367246 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.367257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.367451 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.367723 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.367727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.367749 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.367885 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.367892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.368826 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.368829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.369745 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.369757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.370550 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.370554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.370644 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.370668 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.370743 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.370816 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":179,"thread_start_us":136,"threads_started":1}
I20260810 14:39:38.371021 10443 raft_consensus.cc:482] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:38.371266 10443 raft_consensus.cc:504] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.372016 10220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "536925b43c88439b89c2d85393842b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f4341b7bf46b8964b7d438b028e38" is_pre_election: true
I20260810 14:39:38.372237 10220 raft_consensus.cc:2454] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536925b43c88439b89c2d85393842b91 in term 0.
I20260810 14:39:38.372557 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.366807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:57431 (local address 127.21.43.65:34119)
0810 14:39:38.366925 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.366930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.367353 (+ 423us) server_negotiation.cc:407] Connection header received
0810 14:39:38.367577 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.367580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.367624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.367718 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.368034 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.368700 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.369894 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.370405 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.370533 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.370578 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.372291 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.372403 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:39:38.372686 10443 leader_election.cc:288] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08238d548c6b40be8f9c4838b6eaed32 (127.21.43.65:34119), 536925b43c88439b89c2d85393842b91 (127.21.43.67:45039)
I20260810 14:39:38.372776 10291 leader_election.cc:302] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536925b43c88439b89c2d85393842b91, eb6f4341b7bf46b8964b7d438b028e38; no voters:
I20260810 14:39:38.373350 10441 raft_consensus.cc:2781] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:38.373452 10441 raft_consensus.cc:482] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:38.373502 10441 raft_consensus.cc:3037] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:38.373438 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "536925b43c88439b89c2d85393842b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08238d548c6b40be8f9c4838b6eaed32" is_pre_election: true
I20260810 14:39:38.373737 10066 raft_consensus.cc:2454] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536925b43c88439b89c2d85393842b91 in term 0.
I20260810 14:39:38.374893 10441 raft_consensus.cc:504] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.375344 10441 leader_election.cc:288] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [CANDIDATE]: Term 1 election: Requested vote from peers 08238d548c6b40be8f9c4838b6eaed32 (127.21.43.65:34119), eb6f4341b7bf46b8964b7d438b028e38 (127.21.43.66:35847)
I20260810 14:39:38.375749 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "536925b43c88439b89c2d85393842b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08238d548c6b40be8f9c4838b6eaed32"
I20260810 14:39:38.375782 10220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "536925b43c88439b89c2d85393842b91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6f4341b7bf46b8964b7d438b028e38"
I20260810 14:39:38.375869 10066 raft_consensus.cc:3037] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:38.375908 10220 raft_consensus.cc:3037] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:38.377215 10220 raft_consensus.cc:2454] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536925b43c88439b89c2d85393842b91 in term 1.
I20260810 14:39:38.377292 10066 raft_consensus.cc:2454] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536925b43c88439b89c2d85393842b91 in term 1.
I20260810 14:39:38.377565 10291 leader_election.cc:302] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536925b43c88439b89c2d85393842b91, eb6f4341b7bf46b8964b7d438b028e38; no voters:
I20260810 14:39:38.377758 10441 raft_consensus.cc:2781] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:38.377796 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.373437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:34903 (local address 127.21.43.67:45039)
0810 14:39:38.373559 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.373563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.373631 (+ 68us) server_negotiation.cc:407] Connection header received
0810 14:39:38.373824 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.373828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.373877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.373957 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.374218 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.374875 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.376416 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.376997 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.377103 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.377181 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.377580 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.377662 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:39:38.377995 10441 raft_consensus.cc:686] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 1 LEADER]: Becoming Leader. State: Replica: 536925b43c88439b89c2d85393842b91, State: Running, Role: LEADER
I20260810 14:39:38.378163 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.372998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45039 (local address 127.21.43.66:34903)
0810 14:39:38.373523 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.373539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.373702 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.373972 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.373975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.373994 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.374122 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.374127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.374988 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.374994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.376262 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.376281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.377293 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.377296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.377401 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.377426 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.377908 (+ 482us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.378000 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260810 14:39:38.378338 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.372844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34119 (local address 127.21.43.66:43895)
0810 14:39:38.372983 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.373000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:38.373083 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:38.374745 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:38.374750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:38.374773 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:38.374898 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.374908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.375865 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.375868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.376955 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:38.376967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.377916 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.377921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:38.378063 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.378087 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:39:38.378164 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:39:38.378228 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 14:39:38.378386 10441 consensus_queue.cc:237] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } }
I20260810 14:39:38.378571 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.373243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:43895 (local address 127.21.43.65:34119)
0810 14:39:38.374511 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.374516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.374530 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:39:38.374569 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.374571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.374609 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.374687 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:38.375059 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.375737 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.377123 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.377679 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.377797 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.377850 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.378189 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.378455 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180}
I20260810 14:39:38.378912 10066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "eb6f4341b7bf46b8964b7d438b028e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08238d548c6b40be8f9c4838b6eaed32" is_pre_election: true
I20260810 14:39:38.378976 10364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606442b7a2b74aac9bb679f5599fc65b" candidate_uuid: "eb6f4341b7bf46b8964b7d438b028e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536925b43c88439b89c2d85393842b91" is_pre_election: true
I20260810 14:39:38.379118 10066 raft_consensus.cc:2379] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6f4341b7bf46b8964b7d438b028e38 in current term 1: Already voted for candidate 536925b43c88439b89c2d85393842b91 in this term.
I20260810 14:39:38.379781 10140 leader_election.cc:302] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6f4341b7bf46b8964b7d438b028e38; no voters: 08238d548c6b40be8f9c4838b6eaed32, 536925b43c88439b89c2d85393842b91
I20260810 14:39:38.380533 10443 raft_consensus.cc:2726] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 14:39:38.381898 9902 catalog_manager.cc:5495] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 536925b43c88439b89c2d85393842b91 (127.21.43.67). New cstate: current_term: 1 leader_uuid: "536925b43c88439b89c2d85393842b91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536925b43c88439b89c2d85393842b91" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 } health_report { overall_health: UNKNOWN } } }
I20260810 14:39:38.419731 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.416226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.21.43.67:45039)
0810 14:39:38.416361 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:38.416365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:38.416379 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:39:38.416450 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:38.416452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:38.416509 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:38.416607 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:38.416877 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.417379 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.418237 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:38.418451 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:38.418533 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:38.418818 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:38.419297 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:38.419420 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:38.419557 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 14:39:38.419591 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:39:38.426050 10220 raft_consensus.cc:1261] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 1 FOLLOWER]: Refusing update from remote peer 536925b43c88439b89c2d85393842b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:38.426020 10066 raft_consensus.cc:1261] T 606442b7a2b74aac9bb679f5599fc65b P 08238d548c6b40be8f9c4838b6eaed32 [term 1 FOLLOWER]: Refusing update from remote peer 536925b43c88439b89c2d85393842b91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:38.426752 10441 consensus_queue.cc:1035] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08238d548c6b40be8f9c4838b6eaed32" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:38.427023 10448 consensus_queue.cc:1035] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6f4341b7bf46b8964b7d438b028e38" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:38.435262 10454 mvcc.cc:205] Tried to move back new op lower bound from 7316982900425043968 to 7316982900241629184. Current Snapshot: MvccSnapshot[applied={T|T < 7316982900425043968}]
I20260810 14:39:38.435848 10453 mvcc.cc:205] Tried to move back new op lower bound from 7316982900425043968 to 7316982900241629184. Current Snapshot: MvccSnapshot[applied={T|T < 7316982900425043968}]
I20260810 14:39:38.443753 9902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:38.443959 9902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:38.446453 9902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c7af0c44357443495e60b671612405b: Sending DeleteTablet for 3 replicas of tablet 606442b7a2b74aac9bb679f5599fc65b
I20260810 14:39:38.447336 10343 tablet_service.cc:1514] Processing DeleteTablet for tablet 606442b7a2b74aac9bb679f5599fc65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:38 UTC) from {username='slave'} at 127.0.0.1:57046
I20260810 14:39:38.447752 10194 tablet_service.cc:1514] Processing DeleteTablet for tablet 606442b7a2b74aac9bb679f5599fc65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:38 UTC) from {username='slave'} at 127.0.0.1:42988
I20260810 14:39:38.447930 10043 tablet_service.cc:1514] Processing DeleteTablet for tablet 606442b7a2b74aac9bb679f5599fc65b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:38 UTC) from {username='slave'} at 127.0.0.1:60202
I20260810 14:39:38.448028 10463 tablet_replica.cc:330] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: stopping tablet replica
I20260810 14:39:38.448459 10464 tablet_replica.cc:330] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: stopping tablet replica
I20260810 14:39:38.448468 10463 raft_consensus.cc:2229] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:39:38.448640 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9970
I20260810 14:39:38.448779 10464 raft_consensus.cc:2229] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:38.449148 10464 raft_consensus.cc:2258] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:38.449167 10463 raft_consensus.cc:2258] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:38.451628 10464 ts_tablet_manager.cc:1905] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:38.451684 10463 ts_tablet_manager.cc:1905] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:38.454494 10464 ts_tablet_manager.cc:1918] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:38.454604 10464 log.cc:1196] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/606442b7a2b74aac9bb679f5599fc65b
I20260810 14:39:38.454932 10464 ts_tablet_manager.cc:1939] T 606442b7a2b74aac9bb679f5599fc65b P eb6f4341b7bf46b8964b7d438b028e38: Deleting consensus metadata
I20260810 14:39:38.455446 10463 ts_tablet_manager.cc:1918] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:38.455555 10463 log.cc:1196] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/606442b7a2b74aac9bb679f5599fc65b
I20260810 14:39:38.455902 10463 ts_tablet_manager.cc:1939] T 606442b7a2b74aac9bb679f5599fc65b P 536925b43c88439b89c2d85393842b91: Deleting consensus metadata
I20260810 14:39:38.457144 9889 catalog_manager.cc:4841] TS 536925b43c88439b89c2d85393842b91 (127.21.43.67:45039): tablet 606442b7a2b74aac9bb679f5599fc65b (table test-table [id=c1bc68e77c2b40889269e26763d46dea]) successfully deleted
I20260810 14:39:38.457144 9886 catalog_manager.cc:4841] TS eb6f4341b7bf46b8964b7d438b028e38 (127.21.43.66:35847): tablet 606442b7a2b74aac9bb679f5599fc65b (table test-table [id=c1bc68e77c2b40889269e26763d46dea]) successfully deleted
W20260810 14:39:38.462617 9888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:34119 (error 108)
I20260810 14:39:38.462740 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10121
I20260810 14:39:38.465734 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:38.465252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:34119 (local address 127.0.0.1:60206)
0810 14:39:38.465518 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:38.465601 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260810 14:39:38.465963 9888 catalog_manager.cc:4584] TS 08238d548c6b40be8f9c4838b6eaed32 (127.21.43.65:34119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 606442b7a2b74aac9bb679f5599fc65b: Network error: Client connection negotiation failed: client connection to 127.21.43.65:34119: connect: Connection refused (error 111)
I20260810 14:39:38.472739 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10276
I20260810 14:39:38.481482 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 9867
2026-08-10T14:39:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:38 dist-test-slave-chsk krb5kdc[10470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:38 dist-test-slave-chsk krb5kdc[10470](info): set up 2 sockets
Aug 10 14:39:38 dist-test-slave-chsk krb5kdc[10470](info): commencing operation
krb5kdc: starting...
W20260810 14:39:39.589854 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:39 dist-test-slave-chsk krb5kdc[10470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:39:39.718576 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35313
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:35313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:39.837139 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:39.837441 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:39.837517 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:39.841535 10483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:39.841625 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:39.841665 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:39.841703 10483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:39.841739 10483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:39.847079 10483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:35313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:35313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:39.848163 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:39.849208 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:39.857759 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:39.857769 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:39.857743 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:39.857932 10483 server_base.cc:1034] running on GCE node
I20260810 14:39:39.858654 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:39.859794 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:39.860982 10483 hybrid_clock.cc:648] HybridClock initialized: now 1786372779860966 us; error 33 us; skew 500 ppm
Aug 10 14:39:39 dist-test-slave-chsk krb5kdc[10470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372779, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:39.863794 10483 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:39:39.864717 10483 webserver.cc:460] Webserver started at http://127.21.43.126:45567/ using document root <none> and password file <none>
I20260810 14:39:39.865289 10483 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:39.865384 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:39.865624 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:39.867718 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85"
format_stamp: "Formatted at 2026-08-10 14:39:39 on dist-test-slave-chsk"
server_key: "8056e4bba457f169f877909f558cf057"
server_key_iv: "49cea72f804a3755173433c0acb3b5e8"
server_key_version: "encryptionkey@0"
I20260810 14:39:39.868290 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85"
format_stamp: "Formatted at 2026-08-10 14:39:39 on dist-test-slave-chsk"
server_key: "8056e4bba457f169f877909f558cf057"
server_key_iv: "49cea72f804a3755173433c0acb3b5e8"
server_key_version: "encryptionkey@0"
I20260810 14:39:39.872133 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:39:39.874475 10499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:39.875784 10483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:39.875936 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85"
format_stamp: "Formatted at 2026-08-10 14:39:39 on dist-test-slave-chsk"
server_key: "8056e4bba457f169f877909f558cf057"
server_key_iv: "49cea72f804a3755173433c0acb3b5e8"
server_key_version: "encryptionkey@0"
I20260810 14:39:39.876067 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:39.885115 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:39.887892 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:39.888103 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:39.895494 10483 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:35313
I20260810 14:39:39.895521 10556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:35313 every 8 connection(s)
I20260810 14:39:39.896445 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:39.899127 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:39.904594 10557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: Bootstrap starting.
I20260810 14:39:39.905587 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10483
I20260810 14:39:39.905737 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:39.906107 21677 external_mini_cluster.cc:1428] Setting key aa7cce918e7ddb43d25dbab57fa6da7d
I20260810 14:39:39.907727 10557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:39.908617 10557 log.cc:824] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:39.910885 10557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: No bootstrap required, opened a new log
I20260810 14:39:39.915118 10557 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } }
I20260810 14:39:39.915427 10557 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 14:39:39 dist-test-slave-chsk krb5kdc[10470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:39.915503 10557 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebae46b8d5eb4d11a1d78e1d31aebf85, State: Initialized, Role: FOLLOWER
I20260810 14:39:39.915946 10557 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } }
I20260810 14:39:39.916115 10557 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:39.916188 10557 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:39.916294 10557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:39.917619 10557 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } }
I20260810 14:39:39.917999 10557 leader_election.cc:302] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebae46b8d5eb4d11a1d78e1d31aebf85; no voters:
I20260810 14:39:39.918378 10557 leader_election.cc:288] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:39.918522 10563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:39.918781 10563 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [term 1 LEADER]: Becoming Leader. State: Replica: ebae46b8d5eb4d11a1d78e1d31aebf85, State: Running, Role: LEADER
I20260810 14:39:39.919214 10563 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } }
I20260810 14:39:39.919737 10557 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:39.922113 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:39.907326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.21.43.126:35313)
0810 14:39:39.907761 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:39.907782 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:39.907824 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:39:39.908706 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:39.908730 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:39.909139 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:39.909451 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:39.910152 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:39.911585 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:39.912435 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:39.913164 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:39.913318 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:39.916129 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:39.916139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:39.916157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:39.916194 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:39.918201 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:39.918806 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:39.918813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:39.918820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:39.918968 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:39.919358 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:39.919361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:39.919363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:39.919875 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:39.920035 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:39.920567 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 14:39:39.920901 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
I20260810 14:39:39.922744 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebae46b8d5eb4d11a1d78e1d31aebf85. Latest consensus state: current_term: 1 leader_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } } }
I20260810 14:39:39.922912 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:39.923180 10564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebae46b8d5eb4d11a1d78e1d31aebf85" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 35313 } } }
I20260810 14:39:39.923292 10564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85 [sys.catalog]: This master's current role is: LEADER
W20260810 14:39:39.925689 10579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:39:39.925779 10579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:39:39.925887 10581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:39.926623 10581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:39.932840 10581 catalog_manager.cc:1347] Generated new cluster ID: 0b79578a7d9e44b4832160cdf23b06d4
I20260810 14:39:39.932920 10581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:39.940016 10581 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:39.940878 10581 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:39.946122 10581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: Generated new TSK 0
I20260810 14:39:39.946761 10581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.43.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
Entry for principal kudu/127.21.43.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab.
I20260810 14:39:39.983744 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:40.112876 10587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:40.113179 10587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:40.113226 10587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:40.117295 10587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:40.117362 10587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:40.117422 10587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:40.123013 10587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.65.keytab
--principal=kudu/127.21.43.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:40.124171 10587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:40.125187 10587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:40.134994 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:40.135057 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:40.135212 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:40.135241 10587 server_base.cc:1034] running on GCE node
I20260810 14:39:40.135818 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:40.136628 10587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:40.137835 10587 hybrid_clock.cc:648] HybridClock initialized: now 1786372780137795 us; error 48 us; skew 500 ppm
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:40.140981 10587 init.cc:377] Logged in from keytab as kudu/127.21.43.65@KRBTEST.COM (short username kudu)
I20260810 14:39:40.142038 10587 webserver.cc:460] Webserver started at http://127.21.43.65:40163/ using document root <none> and password file <none>
I20260810 14:39:40.142669 10587 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:40.142766 10587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:40.143011 10587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:40.145247 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "53e23f615d154c0593855cf908cb83f7"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "ebd169782c0aac4a7abbdc09939235b5"
server_key_iv: "241f42489531acc5bcb5f2219f627ec1"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.145841 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "53e23f615d154c0593855cf908cb83f7"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "ebd169782c0aac4a7abbdc09939235b5"
server_key_iv: "241f42489531acc5bcb5f2219f627ec1"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.149778 10587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:40.152352 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.153581 10587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:40.153720 10587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "53e23f615d154c0593855cf908cb83f7"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "ebd169782c0aac4a7abbdc09939235b5"
server_key_iv: "241f42489531acc5bcb5f2219f627ec1"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.153841 10587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:40.174189 10587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:40.177501 10587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:40.177744 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:40.178436 10587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:40.179482 10587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:40.179555 10587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.179636 10587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:40.179678 10587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.189323 10587 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:37757
I20260810 14:39:40.189373 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:37757 every 8 connection(s)
I20260810 14:39:40.190315 10587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:40.191978 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10587
I20260810 14:39:40.192173 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:40.192531 21677 external_mini_cluster.cc:1428] Setting key c1fb4352062086605091f623b9b81f9f
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.65@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.43.66@KRBTEST.COM; defaulting to no policy
I20260810 14:39:40.205989 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.192382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:44913 (local address 127.21.43.126:35313)
0810 14:39:40.192559 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.192564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.193800 (+ 1236us) server_negotiation.cc:407] Connection header received
0810 14:39:40.194819 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.194823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.194879 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.194968 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:40.196319 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.196847 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.198286 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.198711 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.198845 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.201816 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:40.201821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:40.201824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:40.201858 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:40.203756 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.204357 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.204361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.204362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.204458 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.204779 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.204782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.204783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.205057 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:40.205138 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.205694 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.205816 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.66@KRBTEST.COM" created.
I20260810 14:39:40.207002 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.192848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35313 (local address 127.21.43.65:44913)
0810 14:39:40.193647 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.193705 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.194604 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.195194 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.195204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.195590 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:40.196120 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.196144 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.196992 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.196999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.198148 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.198158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.198873 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.198875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.199016 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.199776 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:40.199793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:40.201618 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:40.203885 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.203890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.203904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.204250 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.204553 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.204555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.204557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.204696 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.205189 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:40.205194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:40.205456 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.205637 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.205956 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":150,"threads_started":1}
I20260810 14:39:40.208420 10732 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35313
I20260810 14:39:40.208700 10732 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:40.209357 10732 heartbeater.cc:511] Master 127.21.43.126:35313 requested a full tablet report, sending...
I20260810 14:39:40.211001 10518 ts_manager.cc:194] Registered new tserver with Master: 53e23f615d154c0593855cf908cb83f7 (127.21.43.65:37757)
I20260810 14:39:40.212230 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.65@KRBTEST.COM'} at 127.21.43.65:44913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
Entry for principal kudu/127.21.43.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab.
I20260810 14:39:40.226279 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:40.345589 10737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:40.345912 10737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:40.345988 10737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:40.350157 10737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:40.350266 10737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:40.350366 10737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:40.356007 10737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.66.keytab
--principal=kudu/127.21.43.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:40.357302 10737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:40.358449 10737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:40.368233 10745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:40.368171 10744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:40.368192 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:40.368502 10737 server_base.cc:1034] running on GCE node
I20260810 14:39:40.369045 10737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:40.369730 10737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:40.370927 10737 hybrid_clock.cc:648] HybridClock initialized: now 1786372780370914 us; error 28 us; skew 500 ppm
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:40.374007 10737 init.cc:377] Logged in from keytab as kudu/127.21.43.66@KRBTEST.COM (short username kudu)
I20260810 14:39:40.374943 10737 webserver.cc:460] Webserver started at http://127.21.43.66:35691/ using document root <none> and password file <none>
I20260810 14:39:40.375608 10737 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:40.375706 10737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:40.375950 10737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:40.378093 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "b5cdbaf617834db893c993d750bd2dda"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "63a01065a79e4af6e26e778529148e99"
server_key_iv: "ef32eb421d061cc54cad083e67b85767"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.378695 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "b5cdbaf617834db893c993d750bd2dda"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "63a01065a79e4af6e26e778529148e99"
server_key_iv: "ef32eb421d061cc54cad083e67b85767"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.382658 10737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:40.385370 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.386584 10737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:40.386730 10737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "b5cdbaf617834db893c993d750bd2dda"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "63a01065a79e4af6e26e778529148e99"
server_key_iv: "ef32eb421d061cc54cad083e67b85767"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.386847 10737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:40.403934 10737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:40.407538 10737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:40.407763 10737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:40.408409 10737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:40.409309 10737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:40.409379 10737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.409448 10737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:40.409477 10737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.418660 10737 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:34671
I20260810 14:39:40.418687 10882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:34671 every 8 connection(s)
I20260810 14:39:40.419679 10737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:40.424098 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10737
I20260810 14:39:40.424232 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:40.424541 21677 external_mini_cluster.cc:1428] Setting key 498a3a4f8db460dcc8445daf033ea4b3
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.66@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:40.435312 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.421848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:50379 (local address 127.21.43.126:35313)
0810 14:39:40.422156 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.422160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.423432 (+ 1272us) server_negotiation.cc:407] Connection header received
0810 14:39:40.424624 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.424630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.424701 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.424778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:40.426033 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.426560 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.427834 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.428087 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.428211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.431199 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:40.431204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:40.431209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:40.431241 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:40.432790 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.433467 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.433469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.433471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.433549 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.433986 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.433988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.433989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.434183 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:40.434260 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.434992 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.435109 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
WARNING: no policy specified for kudu/127.21.43.67@KRBTEST.COM; defaulting to no policy
I20260810 14:39:40.436383 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.422369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35313 (local address 127.21.43.66:50379)
0810 14:39:40.423234 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.423308 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.424372 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.424996 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.425014 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.425392 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:40.425881 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.425894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.426704 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.426713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.427683 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.427693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.428284 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.428290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.428449 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.429108 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:40.429126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:40.430940 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:40.432927 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.432936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.432955 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.433356 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.433657 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.433663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.433667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.433882 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.434303 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:40.434310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:40.434701 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.434963 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.435396 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":119,"threads_started":1}
I20260810 14:39:40.437677 10883 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35313
I20260810 14:39:40.438021 10883 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.67@KRBTEST.COM" created.
I20260810 14:39:40.438572 10883 heartbeater.cc:511] Master 127.21.43.126:35313 requested a full tablet report, sending...
I20260810 14:39:40.439776 10518 ts_manager.cc:194] Registered new tserver with Master: b5cdbaf617834db893c993d750bd2dda (127.21.43.66:34671)
I20260810 14:39:40.440524 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.66@KRBTEST.COM'} at 127.21.43.66:50379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
Entry for principal kudu/127.21.43.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab.
I20260810 14:39:40.456375 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:40.574322 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:40.574667 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:40.574750 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:40.579046 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:40.579144 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:40.579264 10888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:40.584929 10888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.67.keytab
--principal=kudu/127.21.43.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:40.586122 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:40.587246 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:40.597479 10898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:40.597550 10895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:40.597536 10888 server_base.cc:1034] running on GCE node
W20260810 14:39:40.597484 10896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:40.598136 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:40.598845 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:40.600040 10888 hybrid_clock.cc:648] HybridClock initialized: now 1786372780600005 us; error 49 us; skew 500 ppm
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:40.603132 10888 init.cc:377] Logged in from keytab as kudu/127.21.43.67@KRBTEST.COM (short username kudu)
I20260810 14:39:40.604166 10888 webserver.cc:460] Webserver started at http://127.21.43.67:33865/ using document root <none> and password file <none>
I20260810 14:39:40.604754 10888 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:40.604818 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:40.605065 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:40.607239 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "653fa703ac9c768a2de07a0d4471873e"
server_key_iv: "fa3a74b38ad77b2753a7bc635ef53051"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.607852 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "653fa703ac9c768a2de07a0d4471873e"
server_key_iv: "fa3a74b38ad77b2753a7bc635ef53051"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.611872 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:39:40.614475 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.615837 10888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:39:40.615990 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490"
format_stamp: "Formatted at 2026-08-10 14:39:40 on dist-test-slave-chsk"
server_key: "653fa703ac9c768a2de07a0d4471873e"
server_key_iv: "fa3a74b38ad77b2753a7bc635ef53051"
server_key_version: "encryptionkey@0"
I20260810 14:39:40.616114 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:40.632769 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:40.636067 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:40.636307 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:40.637009 10888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:40.638056 10888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:40.638129 10888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.638199 10888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:40.638243 10888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:40.647606 10888 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:45943
I20260810 14:39:40.647631 11030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:45943 every 8 connection(s)
I20260810 14:39:40.648552 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:40.654680 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 10888
I20260810 14:39:40.654856 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:40.655305 21677 external_mini_cluster.cc:1428] Setting key 4f158d2986b65ca007ca50276e5bad14
Aug 10 14:39:40 dist-test-slave-chsk krb5kdc[10470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372780, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.67@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:39:40.662725 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.650481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:33901 (local address 127.21.43.126:35313)
0810 14:39:40.650639 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.650643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.651727 (+ 1084us) server_negotiation.cc:407] Connection header received
0810 14:39:40.652661 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.652665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.652729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.652861 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:40.654057 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.654607 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.656139 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.656477 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.656583 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.659432 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:40.659435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:40.659437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:40.659462 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:40.660809 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.661302 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.661304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.661306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.661379 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.661676 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.661678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.661679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.661858 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:40.661931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.662472 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.662596 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:39:40.663650 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.650860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:35313 (local address 127.21.43.67:33901)
0810 14:39:40.651576 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.651618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.652444 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.653038 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.653045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.653425 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:40.653905 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.653917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.654752 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.654757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.656012 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.656026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.656654 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.656658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.656859 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.657557 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:39:40.657574 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:40.659277 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:40.660920 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.660925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.660937 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.661206 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.661455 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:39:40.661457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:39:40.661458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:39:40.661591 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:39:40.661943 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:40.661948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:40.662205 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.662359 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.662623 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":113,"threads_started":1}
I20260810 14:39:40.664891 11033 heartbeater.cc:347] Connected to a master server at 127.21.43.126:35313
I20260810 14:39:40.665144 11033 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:40.665664 11033 heartbeater.cc:511] Master 127.21.43.126:35313 requested a full tablet report, sending...
I20260810 14:39:40.666738 10518 ts_manager.cc:194] Registered new tserver with Master: b87fddb97ceb4f5f9fc84c6a43fb4490 (127.21.43.67:45943)
I20260810 14:39:40.667393 10518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.43.67@KRBTEST.COM'} at 127.21.43.67:33901
I20260810 14:39:40.669468 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:40.678929 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.671611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.21.43.126:35313)
0810 14:39:40.671764 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.671769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.671970 (+ 201us) server_negotiation.cc:407] Connection header received
0810 14:39:40.672061 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.672064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.672106 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.672174 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:40.672860 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.673377 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.674193 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.674403 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.674485 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.675959 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:40.675962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:40.675965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:40.675987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:40.677419 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.677808 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.677810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.677812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.677884 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:40.678110 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:40.678112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:40.678113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:40.678331 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:40.678435 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.678648 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.678747 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:39:40.682027 10518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:40.685302 10518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:40.700379 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.694967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:37757 (local address 127.0.0.1:57816)
0810 14:39:40.695493 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.695510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.695602 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.696317 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.696320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.696342 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:40.696453 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.696460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.697519 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.697526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.698842 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.698853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.699800 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.699804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.699912 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.699950 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.700092 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.700236 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":92,"threads_started":1}
I20260810 14:39:40.700392 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.695134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.21.43.65:37757)
0810 14:39:40.695829 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.695844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.695889 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:39:40.695964 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.695969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.696165 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.696319 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:40.696596 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.697380 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.698988 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.699669 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.699798 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.699960 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.700124 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.700224 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":87,"threads_started":1}
I20260810 14:39:40.700754 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.694194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34671 (local address 127.0.0.1:46954)
0810 14:39:40.695204 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.695261 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.695472 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.696254 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.696263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.696291 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:40.696433 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.696444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.697952 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.697955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.699050 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.699061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.700333 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.700337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.700434 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.700472 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.700536 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.700609 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":87,"threads_started":1}
I20260810 14:39:40.701269 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.694272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.21.43.66:34671)
0810 14:39:40.694982 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.694995 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.695348 (+ 353us) server_negotiation.cc:407] Connection header received
0810 14:39:40.695654 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.695660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.695875 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.696008 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:40.696565 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.697802 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.699215 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.700040 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.700147 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.700271 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.701023 (+ 752us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.701126 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":126,"threads_started":1}
I20260810 14:39:40.702086 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.694268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45943 (local address 127.0.0.1:53388)
0810 14:39:40.695342 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.695355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.695484 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.696406 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.696408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.696425 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:40.696508 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.696517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.699231 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.699234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.700348 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.700356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.701677 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.701681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.701775 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.701797 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.701875 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.701948 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":138,"threads_started":1}
I20260810 14:39:40.702176 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.694460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.21.43.67:45943)
0810 14:39:40.695195 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.695206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.695430 (+ 224us) server_negotiation.cc:407] Connection header received
0810 14:39:40.695885 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.695898 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.696079 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.696194 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:40.696675 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.697800 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.700486 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.701312 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.701405 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.701567 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.701925 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.702036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":182,"threads_started":1}
I20260810 14:39:40.704107 10658 tablet_service.cc:1467] Processing CreateTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 (DEFAULT_TABLE table=test-table [id=4d71d9c853214623a6fb390258c534cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:40.705353 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20205aa8484c43919c9ac4efd2fe2918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:40.705027 10960 tablet_service.cc:1467] Processing CreateTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 (DEFAULT_TABLE table=test-table [id=4d71d9c853214623a6fb390258c534cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:40.706067 10960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20205aa8484c43919c9ac4efd2fe2918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:40.707370 10811 tablet_service.cc:1467] Processing CreateTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 (DEFAULT_TABLE table=test-table [id=4d71d9c853214623a6fb390258c534cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:40.708252 10811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20205aa8484c43919c9ac4efd2fe2918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:40.713106 11055 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: Bootstrap starting.
I20260810 14:39:40.713939 11056 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Bootstrap starting.
I20260810 14:39:40.714581 11057 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Bootstrap starting.
I20260810 14:39:40.715678 11055 tablet_bootstrap.cc:654] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:40.716241 11056 tablet_bootstrap.cc:654] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:40.716490 11055 log.cc:824] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:40.717346 11057 tablet_bootstrap.cc:654] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:40.717608 11056 log.cc:824] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:40.718195 11057 log.cc:824] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:40.719342 11055 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: No bootstrap required, opened a new log
I20260810 14:39:40.719564 11055 ts_tablet_manager.cc:1397] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:39:40.719722 11056 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: No bootstrap required, opened a new log
I20260810 14:39:40.719935 11056 ts_tablet_manager.cc:1397] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:39:40.720408 11057 tablet_bootstrap.cc:492] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: No bootstrap required, opened a new log
I20260810 14:39:40.720585 11057 ts_tablet_manager.cc:1397] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:39:40.723924 11056 raft_consensus.cc:348] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.723744 11055 raft_consensus.cc:348] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.724241 11055 raft_consensus.cc:374] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:40.724336 11055 raft_consensus.cc:729] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e23f615d154c0593855cf908cb83f7, State: Initialized, Role: FOLLOWER
I20260810 14:39:40.724385 11056 raft_consensus.cc:374] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:40.724417 11057 raft_consensus.cc:348] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.724534 11056 raft_consensus.cc:729] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87fddb97ceb4f5f9fc84c6a43fb4490, State: Initialized, Role: FOLLOWER
I20260810 14:39:40.724747 11057 raft_consensus.cc:374] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:40.724817 11057 raft_consensus.cc:729] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdbaf617834db893c993d750bd2dda, State: Initialized, Role: FOLLOWER
I20260810 14:39:40.724885 11055 consensus_queue.cc:260] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.725049 11056 consensus_queue.cc:260] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.725157 11057 consensus_queue.cc:260] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.725860 10883 heartbeater.cc:503] Master 127.21.43.126:35313 was elected leader, sending a full tablet report...
I20260810 14:39:40.726086 11033 heartbeater.cc:503] Master 127.21.43.126:35313 was elected leader, sending a full tablet report...
I20260810 14:39:40.726114 11055 ts_tablet_manager.cc:1428] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 14:39:40.726419 10732 heartbeater.cc:503] Master 127.21.43.126:35313 was elected leader, sending a full tablet report...
I20260810 14:39:40.727015 11056 ts_tablet_manager.cc:1428] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260810 14:39:40.727351 11057 ts_tablet_manager.cc:1428] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:39:40.788367 11062 raft_consensus.cc:482] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:40.788601 11062 raft_consensus.cc:504] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.789764 11062 leader_election.cc:288] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5cdbaf617834db893c993d750bd2dda (127.21.43.66:34671), b87fddb97ceb4f5f9fc84c6a43fb4490 (127.21.43.67:45943)
I20260810 14:39:40.795358 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.790220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:33633 (local address 127.21.43.66:34671)
0810 14:39:40.790364 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.790369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.790770 (+ 401us) server_negotiation.cc:407] Connection header received
0810 14:39:40.790854 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.790857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.790917 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.791004 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:40.791489 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.792288 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.793982 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.794614 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.794746 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.794804 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.795121 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.795218 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:39:40.795358 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.790245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:40305 (local address 127.21.43.67:45943)
0810 14:39:40.790410 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.790415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.790685 (+ 270us) server_negotiation.cc:407] Connection header received
0810 14:39:40.790855 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.790857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.790912 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.791003 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:40.791705 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.792413 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.793981 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.794614 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.794746 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.794803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.795121 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.795225 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:39:40.795436 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.790084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:45943 (local address 127.21.43.65:40305)
0810 14:39:40.790562 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.790580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.790734 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.791030 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.791033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.791061 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:40.791373 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.791383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.792531 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.792534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.793844 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.793857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.794751 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.794755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.794929 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.795009 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.795089 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.795235 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":190,"spinlock_wait_cycles":4352,"thread_start_us":130,"threads_started":1}
I20260810 14:39:40.795439 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.790084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:34671 (local address 127.21.43.65:33633)
0810 14:39:40.790673 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.790687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:40.790771 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:40.791030 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:40.791033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:40.791070 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:40.791379 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.791384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.792391 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.792395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.793844 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:40.793857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.794751 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.794755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:40.794961 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.795013 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 14:39:40.795089 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:39:40.795232 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":407,"spinlock_wait_cycles":94592,"thread_start_us":143,"threads_started":1}
I20260810 14:39:40.796160 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20205aa8484c43919c9ac4efd2fe2918" candidate_uuid: "53e23f615d154c0593855cf908cb83f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" is_pre_election: true
I20260810 14:39:40.796205 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20205aa8484c43919c9ac4efd2fe2918" candidate_uuid: "53e23f615d154c0593855cf908cb83f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cdbaf617834db893c993d750bd2dda" is_pre_election: true
I20260810 14:39:40.796499 10981 raft_consensus.cc:2454] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e23f615d154c0593855cf908cb83f7 in term 0.
I20260810 14:39:40.796504 10833 raft_consensus.cc:2454] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53e23f615d154c0593855cf908cb83f7 in term 0.
I20260810 14:39:40.797001 10608 leader_election.cc:302] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53e23f615d154c0593855cf908cb83f7, b5cdbaf617834db893c993d750bd2dda, b87fddb97ceb4f5f9fc84c6a43fb4490; no voters:
I20260810 14:39:40.797319 11062 raft_consensus.cc:2781] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:40.797401 11062 raft_consensus.cc:482] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:40.797433 11062 raft_consensus.cc:3037] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:40.798509 11062 raft_consensus.cc:504] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.798898 11062 leader_election.cc:288] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [CANDIDATE]: Term 1 election: Requested vote from peers b5cdbaf617834db893c993d750bd2dda (127.21.43.66:34671), b87fddb97ceb4f5f9fc84c6a43fb4490 (127.21.43.67:45943)
I20260810 14:39:40.799289 10833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20205aa8484c43919c9ac4efd2fe2918" candidate_uuid: "53e23f615d154c0593855cf908cb83f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cdbaf617834db893c993d750bd2dda"
I20260810 14:39:40.799324 10981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20205aa8484c43919c9ac4efd2fe2918" candidate_uuid: "53e23f615d154c0593855cf908cb83f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490"
I20260810 14:39:40.799396 10833 raft_consensus.cc:3037] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:40.799465 10981 raft_consensus.cc:3037] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:40.800465 10833 raft_consensus.cc:2454] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e23f615d154c0593855cf908cb83f7 in term 1.
I20260810 14:39:40.800580 10981 raft_consensus.cc:2454] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53e23f615d154c0593855cf908cb83f7 in term 1.
I20260810 14:39:40.800829 10608 leader_election.cc:302] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e23f615d154c0593855cf908cb83f7, b5cdbaf617834db893c993d750bd2dda; no voters:
I20260810 14:39:40.801064 11062 raft_consensus.cc:2781] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:40.801344 11062 raft_consensus.cc:686] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 1 LEADER]: Becoming Leader. State: Replica: 53e23f615d154c0593855cf908cb83f7, State: Running, Role: LEADER
I20260810 14:39:40.801735 11062 consensus_queue.cc:237] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } }
I20260810 14:39:40.805051 10517 catalog_manager.cc:5495] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53e23f615d154c0593855cf908cb83f7 (127.21.43.65). New cstate: current_term: 1 leader_uuid: "53e23f615d154c0593855cf908cb83f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53e23f615d154c0593855cf908cb83f7" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 37757 } health_report { overall_health: HEALTHY } } }
I20260810 14:39:40.824793 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.821276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.21.43.65:37757)
0810 14:39:40.821413 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:40.821418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:40.821440 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:39:40.821594 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:40.821597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:40.821665 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:40.821795 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:40.822095 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.822611 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.823486 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:40.823704 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:40.823786 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:40.824051 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:40.824476 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:40.824580 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:40.824626 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 14:39:40.824677 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:39:40.831072 10981 raft_consensus.cc:1261] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 1 FOLLOWER]: Refusing update from remote peer 53e23f615d154c0593855cf908cb83f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:40.831070 10833 raft_consensus.cc:1261] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 1 FOLLOWER]: Refusing update from remote peer 53e23f615d154c0593855cf908cb83f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:40.831929 11067 consensus_queue.cc:1035] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5cdbaf617834db893c993d750bd2dda" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 34671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:40.832196 11062 consensus_queue.cc:1035] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87fddb97ceb4f5f9fc84c6a43fb4490" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:40.840570 11072 mvcc.cc:205] Tried to move back new op lower bound from 7316982910275719168 to 7316982910166675456. Current Snapshot: MvccSnapshot[applied={T|T < 7316982910275719168}]
I20260810 14:39:40.849416 10518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:40.849594 10518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:40.852242 10518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebae46b8d5eb4d11a1d78e1d31aebf85: Sending DeleteTablet for 3 replicas of tablet 20205aa8484c43919c9ac4efd2fe2918
I20260810 14:39:40.853183 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:40 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:57816
I20260810 14:39:40.853135 10960 tablet_service.cc:1514] Processing DeleteTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:40 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:53388
I20260810 14:39:40.853206 10811 tablet_service.cc:1514] Processing DeleteTablet for tablet 20205aa8484c43919c9ac4efd2fe2918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:40 UTC) from {username='kudu', principal='kudu/127.21.43.126@KRBTEST.COM'} at 127.0.0.1:46954
I20260810 14:39:40.853746 11084 tablet_replica.cc:330] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7: stopping tablet replica
I20260810 14:39:40.853888 11085 tablet_replica.cc:330] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: stopping tablet replica
I20260810 14:39:40.854174 11085 raft_consensus.cc:2229] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:40.854174 11084 raft_consensus.cc:2229] T 20205aa8484c43919c9ac4efd2fe2918 P 53e23f615d154c0593855cf908cb83f7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:39:40.854454 11086 tablet_replica.cc:330] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: stopping tablet replica
I20260810 14:39:40.854574 11085 raft_consensus.cc:2258] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:40.854596 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10587
I20260810 14:39:40.854801 11086 raft_consensus.cc:2229] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:40.856022 11086 raft_consensus.cc:2258] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:40.858053 11086 ts_tablet_manager.cc:1905] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:40.858289 11085 ts_tablet_manager.cc:1905] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:40.860589 11085 ts_tablet_manager.cc:1918] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:40.860719 11085 log.cc:1196] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/20205aa8484c43919c9ac4efd2fe2918
I20260810 14:39:40.861038 11085 ts_tablet_manager.cc:1939] T 20205aa8484c43919c9ac4efd2fe2918 P b87fddb97ceb4f5f9fc84c6a43fb4490: Deleting consensus metadata
I20260810 14:39:40.861742 11086 ts_tablet_manager.cc:1918] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:40.861855 11086 log.cc:1196] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/20205aa8484c43919c9ac4efd2fe2918
I20260810 14:39:40.862293 11086 ts_tablet_manager.cc:1939] T 20205aa8484c43919c9ac4efd2fe2918 P b5cdbaf617834db893c993d750bd2dda: Deleting consensus metadata
I20260810 14:39:40.862510 10500 catalog_manager.cc:4841] TS b87fddb97ceb4f5f9fc84c6a43fb4490 (127.21.43.67:45943): tablet 20205aa8484c43919c9ac4efd2fe2918 (table test-table [id=4d71d9c853214623a6fb390258c534cb]) successfully deleted
I20260810 14:39:40.863775 10503 catalog_manager.cc:4841] TS b5cdbaf617834db893c993d750bd2dda (127.21.43.66:34671): tablet 20205aa8484c43919c9ac4efd2fe2918 (table test-table [id=4d71d9c853214623a6fb390258c534cb]) successfully deleted
W20260810 14:39:40.865747 10501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:37757 (error 108)
I20260810 14:39:40.866734 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10737
I20260810 14:39:40.866967 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:40.866609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:37757 (local address 127.0.0.1:57834)
0810 14:39:40.866754 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:40.866858 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:37757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 14:39:40.868834 10501 catalog_manager.cc:4584] TS 53e23f615d154c0593855cf908cb83f7 (127.21.43.65:37757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20205aa8484c43919c9ac4efd2fe2918: Network error: Client connection negotiation failed: client connection to 127.21.43.65:37757: connect: Connection refused (error 111)
I20260810 14:39:40.875502 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10888
I20260810 14:39:40.885468 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 10483
2026-08-10T14:39:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2404 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T14:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:40Z Disabled control of system clock
I20260810 14:39:40.920368 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41553
--webserver_interface=127.21.43.126
--webserver_port=0
--builtin_ntp_servers=127.21.43.84:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:41553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:41.036976 11091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:41.037256 11091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:41.037292 11091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:41.041501 11091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:41.041575 11091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:41.041591 11091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:41.041635 11091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:41.041671 11091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:41.046691 11091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:41553
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:41.047914 11091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:41.049300 11091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:41.058015 11098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.058000 11097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:41.058131 11091 server_base.cc:1034] running on GCE node
W20260810 14:39:41.057978 11100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:41.058856 11091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:41.060132 11091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:41.061355 11091 hybrid_clock.cc:648] HybridClock initialized: now 1786372781061318 us; error 47 us; skew 500 ppm
I20260810 14:39:41.067508 11091 webserver.cc:460] Webserver started at http://127.21.43.126:44205/ using document root <none> and password file <none>
I20260810 14:39:41.068182 11091 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:41.068287 11091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:41.068557 11091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:41.070778 11091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "34144eab83fc4b73bae97f29f4523ed2"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "d6faf9417374407046488acdbeec4884"
server_key_iv: "c35074f97057f49db7707e0599887966"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.071420 11091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "34144eab83fc4b73bae97f29f4523ed2"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "d6faf9417374407046488acdbeec4884"
server_key_iv: "c35074f97057f49db7707e0599887966"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.075307 11091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:39:41.077875 11107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.079087 11091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:41.079265 11091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "34144eab83fc4b73bae97f29f4523ed2"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "d6faf9417374407046488acdbeec4884"
server_key_iv: "c35074f97057f49db7707e0599887966"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.079394 11091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:41.090636 11091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:41.091420 11091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:41.091627 11091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:41.098888 11091 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:41553
I20260810 14:39:41.098892 11163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:41553 every 8 connection(s)
I20260810 14:39:41.099932 11091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:41.103230 11164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:41.107226 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11091
I20260810 14:39:41.107378 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:41.107681 21677 external_mini_cluster.cc:1428] Setting key fcd0d36b595e6a5a6c62a0e794c662ae
I20260810 14:39:41.109576 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: Bootstrap starting.
I20260810 14:39:41.112109 11164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:41.112902 11164 log.cc:824] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:41.114945 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: No bootstrap required, opened a new log
I20260810 14:39:41.117691 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.109226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.21.43.126:41553)
0810 14:39:41.109993 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.110010 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.110055 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:39:41.110726 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.110743 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.111201 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.111565 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:41.112009 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.113095 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.114191 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.114669 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.114847 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.115813 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:41.115818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:41.115831 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:41.116215 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:41.116271 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:41.116290 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:41.116403 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.116523 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.116782 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":149,"threads_started":1}
I20260810 14:39:41.118332 11164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } }
I20260810 14:39:41.118605 11164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:41.118666 11164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34144eab83fc4b73bae97f29f4523ed2, State: Initialized, Role: FOLLOWER
I20260810 14:39:41.119179 11164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } }
I20260810 14:39:41.119319 11164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:41.119403 11164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:41.119519 11164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:41.120643 11164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } }
I20260810 14:39:41.120997 11164 leader_election.cc:302] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34144eab83fc4b73bae97f29f4523ed2; no voters:
I20260810 14:39:41.121311 11164 leader_election.cc:288] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:41.121454 11169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:41.121719 11169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 34144eab83fc4b73bae97f29f4523ed2, State: Running, Role: LEADER
I20260810 14:39:41.122110 11169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } }
I20260810 14:39:41.122670 11164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:41.123224 11169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34144eab83fc4b73bae97f29f4523ed2. Latest consensus state: current_term: 1 leader_uuid: "34144eab83fc4b73bae97f29f4523ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } } }
I20260810 14:39:41.123354 11169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:41.123699 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34144eab83fc4b73bae97f29f4523ed2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144eab83fc4b73bae97f29f4523ed2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41553 } } }
I20260810 14:39:41.123795 11174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:41.123792 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:41.126664 11174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:41.132941 11174 catalog_manager.cc:1347] Generated new cluster ID: 26a1a1abcda84978b4c05620f7a943dc
I20260810 14:39:41.133060 11174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:41.162223 11174 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:41.163311 11174 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:41.169705 11174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: Generated new TSK 0
I20260810 14:39:41.170482 11174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:41.176047 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.65:0
--local_ip_for_outbound_sockets=127.21.43.65
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--builtin_ntp_servers=127.21.43.84:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:39:41.295307 11192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:41.295611 11192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:41.295653 11192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:41.299877 11192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:41.299947 11192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:41.300034 11192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.65
I20260810 14:39:41.305154 11192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.43.65
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.65
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:41.306373 11192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:41.307924 11192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:41.317764 11198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.317840 11201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.317776 11199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:41.318003 11192 server_base.cc:1034] running on GCE node
I20260810 14:39:41.318568 11192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:41.319320 11192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:41.320499 11192 hybrid_clock.cc:648] HybridClock initialized: now 1786372781320478 us; error 28 us; skew 500 ppm
I20260810 14:39:41.322549 11192 webserver.cc:460] Webserver started at http://127.21.43.65:44269/ using document root <none> and password file <none>
I20260810 14:39:41.323212 11192 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:41.323285 11192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:41.323537 11192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:41.325886 11192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data/instance:
uuid: "596a8e2300b549ba8285a11ac8615eb6"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "cd5802d4cafcb789fb9317562d403afd"
server_key_iv: "22c8ad61b12e07f7268008078b7fafba"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.326465 11192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance:
uuid: "596a8e2300b549ba8285a11ac8615eb6"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "cd5802d4cafcb789fb9317562d403afd"
server_key_iv: "22c8ad61b12e07f7268008078b7fafba"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.330288 11192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:41.332828 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.334136 11192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:41.334290 11192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal
uuid: "596a8e2300b549ba8285a11ac8615eb6"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "cd5802d4cafcb789fb9317562d403afd"
server_key_iv: "22c8ad61b12e07f7268008078b7fafba"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.334424 11192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:41.344592 11192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:41.345304 11192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:41.345515 11192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:41.346119 11192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:41.347095 11192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:41.347186 11192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.347265 11192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:41.347316 11192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.356881 11192 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.65:38847
I20260810 14:39:41.356899 11331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.65:38847 every 8 connection(s)
I20260810 14:39:41.357895 11192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/data/info.pb
I20260810 14:39:41.364116 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11192
I20260810 14:39:41.364289 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-0/wal/instance
I20260810 14:39:41.364679 21677 external_mini_cluster.cc:1428] Setting key e77228fee0d69da3d1b93d7c076a10d7
I20260810 14:39:41.366986 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.66:0
--local_ip_for_outbound_sockets=127.21.43.66
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--builtin_ntp_servers=127.21.43.84:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:41.368368 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.359773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:46863 (local address 127.21.43.126:41553)
0810 14:39:41.359918 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.359922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.360969 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 14:39:41.362467 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.362471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.362556 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.362654 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:41.363675 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.364237 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.365811 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.366095 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.366205 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.367744 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:41.367751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:41.367757 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:41.367775 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:41.367809 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:41.367824 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:41.367909 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.368074 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.368153 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:39:41.369378 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.360136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41553 (local address 127.21.43.65:46863)
0810 14:39:41.360809 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.360851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.362234 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.362839 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.362846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.362931 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:41.363521 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.363535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.364446 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.364451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.365669 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.365682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.366284 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.366290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.366483 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.367410 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:41.367413 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:41.367470 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:41.367555 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:41.367921 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:41.367929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:41.367934 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.368043 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.368294 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260810 14:39:41.370735 11332 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41553
I20260810 14:39:41.371001 11332 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:41.371642 11332 heartbeater.cc:511] Master 127.21.43.126:41553 requested a full tablet report, sending...
I20260810 14:39:41.373281 11126 ts_manager.cc:194] Registered new tserver with Master: 596a8e2300b549ba8285a11ac8615eb6 (127.21.43.65:38847)
I20260810 14:39:41.374469 11126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.65:46863
W20260810 14:39:41.483835 11336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:41.484156 11336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:41.484231 11336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:41.488404 11336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:41.488504 11336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:41.488595 11336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.66
I20260810 14:39:41.493667 11336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.43.66
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.66
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:41.494966 11336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:41.496507 11336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:41.506539 11342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.506606 11343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.506583 11345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:41.506860 11336 server_base.cc:1034] running on GCE node
I20260810 14:39:41.507398 11336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:41.508108 11336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:41.509330 11336 hybrid_clock.cc:648] HybridClock initialized: now 1786372781509315 us; error 31 us; skew 500 ppm
I20260810 14:39:41.511644 11336 webserver.cc:460] Webserver started at http://127.21.43.66:33101/ using document root <none> and password file <none>
I20260810 14:39:41.512292 11336 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:41.512395 11336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:41.512650 11336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:41.514909 11336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data/instance:
uuid: "51bfec0c018647bcbe4575d77987689b"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "f615d90cb760a974dac5a3038798eb49"
server_key_iv: "f2324ffd11f47febf8181efc6245e6ce"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.515596 11336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance:
uuid: "51bfec0c018647bcbe4575d77987689b"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "f615d90cb760a974dac5a3038798eb49"
server_key_iv: "f2324ffd11f47febf8181efc6245e6ce"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.519626 11336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:39:41.522316 11352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.523759 11336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:41.523898 11336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal
uuid: "51bfec0c018647bcbe4575d77987689b"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "f615d90cb760a974dac5a3038798eb49"
server_key_iv: "f2324ffd11f47febf8181efc6245e6ce"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.524035 11336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:41.545758 11336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:41.546542 11336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:41.546763 11336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:41.547492 11336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:41.548596 11336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:41.548671 11336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.548748 11336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:41.548799 11336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.557864 11336 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.66:45153
I20260810 14:39:41.557888 11475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.66:45153 every 8 connection(s)
I20260810 14:39:41.558897 11336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/data/info.pb
I20260810 14:39:41.565779 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11336
I20260810 14:39:41.565945 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal/instance
I20260810 14:39:41.566277 21677 external_mini_cluster.cc:1428] Setting key dc3ff3269d4a835ef0ef8929adb2c163
I20260810 14:39:41.568548 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.43.67:0
--local_ip_for_outbound_sockets=127.21.43.67
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--builtin_ntp_servers=127.21.43.84:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:39:41.569337 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.560915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.66:48301 (local address 127.21.43.126:41553)
0810 14:39:41.561068 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.561072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.562200 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 14:39:41.563718 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.563721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.563773 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.563841 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:41.564769 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.565342 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.567002 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.567350 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.567460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.568701 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:41.568706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:41.568713 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:41.568730 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:41.568758 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:41.568772 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:41.568855 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.569030 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.569110 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:39:41.570182 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.561370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41553 (local address 127.21.43.66:48301)
0810 14:39:41.562065 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.562107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.563490 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.564013 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.564019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.564102 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:41.564591 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.564603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.565496 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.565500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.566834 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.566845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.567528 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.567532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.567680 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.568409 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:41.568410 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:41.568454 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:41.568523 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:41.568859 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:41.568868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:41.568875 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.569008 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.569270 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":127,"threads_started":1}
I20260810 14:39:41.571436 11476 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41553
I20260810 14:39:41.571734 11476 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:41.572347 11476 heartbeater.cc:511] Master 127.21.43.126:41553 requested a full tablet report, sending...
I20260810 14:39:41.573549 11126 ts_manager.cc:194] Registered new tserver with Master: 51bfec0c018647bcbe4575d77987689b (127.21.43.66:45153)
I20260810 14:39:41.574465 11126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.66:48301
W20260810 14:39:41.688616 11479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:41.688918 11479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:41.688959 11479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:41.693145 11479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:41.693218 11479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:41.693277 11479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.43.67
I20260810 14:39:41.698261 11479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.43.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.43.67
--webserver_port=0
--tserver_master_addrs=127.21.43.126:41553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.43.67
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:41.699498 11479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:41.700891 11479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:41.710696 11489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.710790 11484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:41.710727 11487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:41.710923 11479 server_base.cc:1034] running on GCE node
I20260810 14:39:41.711414 11479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:41.712083 11479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:41.713250 11479 hybrid_clock.cc:648] HybridClock initialized: now 1786372781713223 us; error 39 us; skew 500 ppm
I20260810 14:39:41.715250 11479 webserver.cc:460] Webserver started at http://127.21.43.67:34243/ using document root <none> and password file <none>
I20260810 14:39:41.715839 11479 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:41.715905 11479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:41.716156 11479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:41.718307 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data/instance:
uuid: "af8cc11f8f9a406482abbb605cb61382"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "49862df9f69fdee07f09f1e1fbcd00fd"
server_key_iv: "cdf6564fed03ceb7adfd79a9a69c82b2"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.718847 11479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance:
uuid: "af8cc11f8f9a406482abbb605cb61382"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "49862df9f69fdee07f09f1e1fbcd00fd"
server_key_iv: "cdf6564fed03ceb7adfd79a9a69c82b2"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.723407 11479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260810 14:39:41.726112 11495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.727679 11479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:41.727811 11479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal
uuid: "af8cc11f8f9a406482abbb605cb61382"
format_stamp: "Formatted at 2026-08-10 14:39:41 on dist-test-slave-chsk"
server_key: "49862df9f69fdee07f09f1e1fbcd00fd"
server_key_iv: "cdf6564fed03ceb7adfd79a9a69c82b2"
server_key_version: "encryptionkey@0"
I20260810 14:39:41.727939 11479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:41.757320 11479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:41.758024 11479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:41.758203 11479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:41.758777 11479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:39:41.759799 11479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:39:41.759850 11479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.759892 11479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:39:41.759907 11479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:41.769328 11479 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.67:44343
I20260810 14:39:41.769354 11619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.67:44343 every 8 connection(s)
I20260810 14:39:41.770292 11479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/data/info.pb
I20260810 14:39:41.777382 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11479
I20260810 14:39:41.777499 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal/instance
I20260810 14:39:41.777796 21677 external_mini_cluster.cc:1428] Setting key 63ac07d3dcb5f4ca5523dbcbd1e72ad7
I20260810 14:39:41.780174 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.772364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.67:50717 (local address 127.21.43.126:41553)
0810 14:39:41.772540 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.772544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.773602 (+ 1058us) server_negotiation.cc:407] Connection header received
0810 14:39:41.775128 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.775131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.775215 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.775326 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:41.776257 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.776797 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.778092 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.778325 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.778441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.779658 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:41.779665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:41.779668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:41.779680 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:41.779699 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:41.779707 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:41.779783 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.779957 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.780027 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:39:41.781066 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.772762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:41553 (local address 127.21.43.67:50717)
0810 14:39:41.773437 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.773478 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.774888 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.775506 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.775512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.775600 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:39:41.776100 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.776113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.776930 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.776935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.777958 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.777969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.778491 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.778495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.778671 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.779357 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:39:41.779359 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:39:41.779402 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:39:41.779470 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:39:41.779798 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:39:41.779809 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:39:41.779822 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.779955 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.780267 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260810 14:39:41.782238 11620 heartbeater.cc:347] Connected to a master server at 127.21.43.126:41553
I20260810 14:39:41.782513 11620 heartbeater.cc:464] Registering TS with master...
I20260810 14:39:41.783088 11620 heartbeater.cc:511] Master 127.21.43.126:41553 requested a full tablet report, sending...
I20260810 14:39:41.784330 11126 ts_manager.cc:194] Registered new tserver with Master: af8cc11f8f9a406482abbb605cb61382 (127.21.43.67:44343)
I20260810 14:39:41.784934 11126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.43.67:50717
I20260810 14:39:41.792001 21677 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:39:41.797746 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.794036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.21.43.126:41553)
0810 14:39:41.794181 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.794185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.794352 (+ 167us) server_negotiation.cc:407] Connection header received
0810 14:39:41.794462 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.794465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.794508 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.794568 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:41.794840 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.795421 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.796269 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.796470 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.796544 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.797371 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:41.797374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:41.797376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:41.797388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:41.797407 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:41.797414 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:41.797473 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.797579 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.797628 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:39:41.800705 11126 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:39:41.803753 11126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:39:41.819093 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.813503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.21.43.67:44343)
0810 14:39:41.814036 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.814049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.814078 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:39:41.814157 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.814162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.814390 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.814553 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:41.814879 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.816096 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.817652 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818324 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818445 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.818671 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.818809 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.818914 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":172,"threads_started":1}
I20260810 14:39:41.819090 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.813030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:44343 (local address 127.0.0.1:36092)
0810 14:39:41.813715 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.813770 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.813903 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.814551 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.814557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.814581 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:41.814731 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.814738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.816226 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.816229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.817506 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.817517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818459 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.818572 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.818642 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.818767 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.818918 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":48,"spinlock_wait_cycles":7424,"thread_start_us":86,"threads_started":1}
I20260810 14:39:41.819751 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.812922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.21.43.66:45153)
0810 14:39:41.814409 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.814419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.814439 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:39:41.814529 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.814534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.814717 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.815159 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:41.815299 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.816540 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.817921 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818685 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818967 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.819173 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.819472 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.819585 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":173,"threads_started":1}
I20260810 14:39:41.819774 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.812765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45153 (local address 127.0.0.1:51102)
0810 14:39:41.813713 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.813770 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.813905 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.814875 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.814878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.814897 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:41.814979 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.814985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.816667 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.816670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.817777 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.817789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818815 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.818949 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.818974 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.819050 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.819125 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":105,"spinlock_wait_cycles":2816,"thread_start_us":104,"threads_started":1}
I20260810 14:39:41.820667 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.812765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38847 (local address 127.0.0.1:58142)
0810 14:39:41.814174 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:41.814188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:41.814276 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:41.814738 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:41.814741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:41.814760 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:41.814861 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.814870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.815936 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.815941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.817506 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:41.817517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818788 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:41.820321 (+ 1526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.820351 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:39:41.820447 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:39:41.820528 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":773,"thread_start_us":92,"threads_started":1}
I20260810 14:39:41.820787 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:41.813017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.21.43.65:38847)
0810 14:39:41.814228 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:41.814238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:41.814259 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:39:41.814413 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:41.814419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:41.814613 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:41.814759 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:41.814995 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.815814 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.817670 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:41.818415 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:41.818529 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:41.818716 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:41.820506 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0810 14:39:41.820609 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":178,"threads_started":1}
I20260810 14:39:41.822566 11404 tablet_service.cc:1467] Processing CreateTablet for tablet 5a595896d0d141809a593338195f7091 (DEFAULT_TABLE table=test-table [id=32bc053dff9548d592125fc9635e4914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:41.822566 11547 tablet_service.cc:1467] Processing CreateTablet for tablet 5a595896d0d141809a593338195f7091 (DEFAULT_TABLE table=test-table [id=32bc053dff9548d592125fc9635e4914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:41.823827 11547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a595896d0d141809a593338195f7091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:41.823822 11404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a595896d0d141809a593338195f7091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:41.830519 11260 tablet_service.cc:1467] Processing CreateTablet for tablet 5a595896d0d141809a593338195f7091 (DEFAULT_TABLE table=test-table [id=32bc053dff9548d592125fc9635e4914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:39:41.831684 11260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a595896d0d141809a593338195f7091. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:41.832125 11639 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Bootstrap starting.
I20260810 14:39:41.832561 11640 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Bootstrap starting.
I20260810 14:39:41.835232 11639 tablet_bootstrap.cc:654] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:41.836177 11639 log.cc:824] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:41.837358 11640 tablet_bootstrap.cc:654] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:41.838141 11640 log.cc:824] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:41.838526 11639 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: No bootstrap required, opened a new log
I20260810 14:39:41.838804 11639 ts_tablet_manager.cc:1397] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 14:39:41.839598 11643 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Bootstrap starting.
I20260810 14:39:41.840831 11640 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: No bootstrap required, opened a new log
I20260810 14:39:41.841099 11640 ts_tablet_manager.cc:1397] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 14:39:41.842087 11643 tablet_bootstrap.cc:654] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:41.842808 11639 raft_consensus.cc:348] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.843009 11643 log.cc:824] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:41.843065 11639 raft_consensus.cc:374] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:41.843119 11639 raft_consensus.cc:729] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8cc11f8f9a406482abbb605cb61382, State: Initialized, Role: FOLLOWER
I20260810 14:39:41.843626 11639 consensus_queue.cc:260] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.844337 11620 heartbeater.cc:503] Master 127.21.43.126:41553 was elected leader, sending a full tablet report...
I20260810 14:39:41.845544 11643 tablet_bootstrap.cc:492] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: No bootstrap required, opened a new log
I20260810 14:39:41.845726 11639 ts_tablet_manager.cc:1428] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260810 14:39:41.845760 11643 ts_tablet_manager.cc:1397] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 14:39:41.848074 11640 raft_consensus.cc:348] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.848316 11640 raft_consensus.cc:374] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:41.848380 11640 raft_consensus.cc:729] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bfec0c018647bcbe4575d77987689b, State: Initialized, Role: FOLLOWER
I20260810 14:39:41.848852 11640 consensus_queue.cc:260] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.849519 11476 heartbeater.cc:503] Master 127.21.43.126:41553 was elected leader, sending a full tablet report...
I20260810 14:39:41.850347 11643 raft_consensus.cc:348] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.850644 11643 raft_consensus.cc:374] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:41.850699 11643 raft_consensus.cc:729] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596a8e2300b549ba8285a11ac8615eb6, State: Initialized, Role: FOLLOWER
I20260810 14:39:41.850842 11640 ts_tablet_manager.cc:1428] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 14:39:41.851178 11643 consensus_queue.cc:260] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:41.851923 11332 heartbeater.cc:503] Master 127.21.43.126:41553 was elected leader, sending a full tablet report...
I20260810 14:39:41.852149 11643 ts_tablet_manager.cc:1428] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 14:39:41.859113 11333 tablet.cc:2369] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:39:42.021421 11622 tablet.cc:2369] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:39:42.060190 11477 tablet.cc:2369] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:39:42.065429 11648 raft_consensus.cc:482] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:39:42.065717 11648 raft_consensus.cc:504] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:42.066941 11648 leader_election.cc:288] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51bfec0c018647bcbe4575d77987689b (127.21.43.66:45153), af8cc11f8f9a406482abbb605cb61382 (127.21.43.67:44343)
I20260810 14:39:42.072623 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.067414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:37919 (local address 127.21.43.66:45153)
0810 14:39:42.067546 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:42.067551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:42.067905 (+ 354us) server_negotiation.cc:407] Connection header received
0810 14:39:42.067997 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:42.068000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:42.068054 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:42.068168 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:42.068584 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.069379 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071209 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.071820 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071962 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:42.072014 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:42.072399 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 14:39:42.072484 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:39:42.072623 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.067414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.43.65:42813 (local address 127.21.43.67:44343)
0810 14:39:42.067546 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:42.067551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:42.067824 (+ 273us) server_negotiation.cc:407] Connection header received
0810 14:39:42.067997 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:42.068000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:42.068054 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:42.068154 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:39:42.068586 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.069379 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071210 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.071831 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071962 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:42.072014 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:42.072399 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 14:39:42.072484 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:39:42.072679 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.067309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.67:44343 (local address 127.21.43.65:42813)
0810 14:39:42.067720 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:42.067739 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:42.067871 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:42.068182 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:42.068185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:42.068204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:42.068462 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:42.068469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.069502 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.069505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:42.071030 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:42.071043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.071972 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:42.072198 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:42.072261 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 14:39:42.072344 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:39:42.072473 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":319,"spinlock_wait_cycles":3456,"thread_start_us":83,"threads_started":1}
I20260810 14:39:42.072679 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.067309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.66:45153 (local address 127.21.43.65:37919)
0810 14:39:42.067818 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:42.067831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:39:42.067917 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:39:42.068182 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:39:42.068185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:39:42.068218 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:39:42.068462 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:42.068469 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.069502 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.069505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:42.071039 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:39:42.071044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.071965 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.071969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:39:42.072217 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:42.072265 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:39:42.072344 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:39:42.072473 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":307,"spinlock_wait_cycles":896,"thread_start_us":129,"threads_started":1}
I20260810 14:39:42.073417 11567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a595896d0d141809a593338195f7091" candidate_uuid: "596a8e2300b549ba8285a11ac8615eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8cc11f8f9a406482abbb605cb61382" is_pre_election: true
I20260810 14:39:42.073712 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a595896d0d141809a593338195f7091" candidate_uuid: "596a8e2300b549ba8285a11ac8615eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bfec0c018647bcbe4575d77987689b" is_pre_election: true
I20260810 14:39:42.073757 11567 raft_consensus.cc:2454] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596a8e2300b549ba8285a11ac8615eb6 in term 0.
I20260810 14:39:42.073994 11424 raft_consensus.cc:2454] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596a8e2300b549ba8285a11ac8615eb6 in term 0.
I20260810 14:39:42.074290 11212 leader_election.cc:302] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596a8e2300b549ba8285a11ac8615eb6, af8cc11f8f9a406482abbb605cb61382; no voters:
I20260810 14:39:42.074584 11648 raft_consensus.cc:2781] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:39:42.074682 11648 raft_consensus.cc:482] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:39:42.074730 11648 raft_consensus.cc:3037] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:42.075917 11648 raft_consensus.cc:504] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:42.076272 11648 leader_election.cc:288] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [CANDIDATE]: Term 1 election: Requested vote from peers 51bfec0c018647bcbe4575d77987689b (127.21.43.66:45153), af8cc11f8f9a406482abbb605cb61382 (127.21.43.67:44343)
I20260810 14:39:42.076602 11424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a595896d0d141809a593338195f7091" candidate_uuid: "596a8e2300b549ba8285a11ac8615eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bfec0c018647bcbe4575d77987689b"
I20260810 14:39:42.076646 11567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a595896d0d141809a593338195f7091" candidate_uuid: "596a8e2300b549ba8285a11ac8615eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8cc11f8f9a406482abbb605cb61382"
I20260810 14:39:42.076746 11424 raft_consensus.cc:3037] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:42.076797 11567 raft_consensus.cc:3037] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:42.077785 11567 raft_consensus.cc:2454] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596a8e2300b549ba8285a11ac8615eb6 in term 1.
I20260810 14:39:42.077788 11424 raft_consensus.cc:2454] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596a8e2300b549ba8285a11ac8615eb6 in term 1.
I20260810 14:39:42.078123 11212 leader_election.cc:302] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596a8e2300b549ba8285a11ac8615eb6, af8cc11f8f9a406482abbb605cb61382; no voters:
I20260810 14:39:42.078321 11648 raft_consensus.cc:2781] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:42.078573 11648 raft_consensus.cc:686] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 1 LEADER]: Becoming Leader. State: Replica: 596a8e2300b549ba8285a11ac8615eb6, State: Running, Role: LEADER
I20260810 14:39:42.078948 11648 consensus_queue.cc:237] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } }
I20260810 14:39:42.082705 11126 catalog_manager.cc:5495] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 596a8e2300b549ba8285a11ac8615eb6 (127.21.43.65). New cstate: current_term: 1 leader_uuid: "596a8e2300b549ba8285a11ac8615eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596a8e2300b549ba8285a11ac8615eb6" member_type: VOTER last_known_addr { host: "127.21.43.65" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 } health_report { overall_health: UNKNOWN } } }
I20260810 14:39:42.112622 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.108471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.21.43.65:38847)
0810 14:39:42.108630 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:42.108635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:42.108651 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:39:42.108754 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:42.108757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:42.108816 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:42.108920 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:39:42.109193 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.109771 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.111024 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:42.111308 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:42.111390 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:42.111700 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:39:42.112122 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:39:42.112215 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:42.112318 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:39:42.112417 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:39:42.118619 11424 raft_consensus.cc:1261] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 1 FOLLOWER]: Refusing update from remote peer 596a8e2300b549ba8285a11ac8615eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:42.118619 11567 raft_consensus.cc:1261] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 1 FOLLOWER]: Refusing update from remote peer 596a8e2300b549ba8285a11ac8615eb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:39:42.119371 11648 consensus_queue.cc:1035] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8cc11f8f9a406482abbb605cb61382" member_type: VOTER last_known_addr { host: "127.21.43.67" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:42.119585 11652 consensus_queue.cc:1035] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bfec0c018647bcbe4575d77987689b" member_type: VOTER last_known_addr { host: "127.21.43.66" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:39:42.130781 11656 mvcc.cc:205] Tried to move back new op lower bound from 7316982915550359552 to 7316982915398529024. Current Snapshot: MvccSnapshot[applied={T|T < 7316982915550359552}]
I20260810 14:39:42.136111 11125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:42.136323 11125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:39:42.139041 11125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34144eab83fc4b73bae97f29f4523ed2: Sending DeleteTablet for 3 replicas of tablet 5a595896d0d141809a593338195f7091
I20260810 14:39:42.139894 11260 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a595896d0d141809a593338195f7091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:42 UTC) from {username='slave'} at 127.0.0.1:58142
I20260810 14:39:42.139901 11404 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a595896d0d141809a593338195f7091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:42 UTC) from {username='slave'} at 127.0.0.1:51102
I20260810 14:39:42.140105 11547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a595896d0d141809a593338195f7091 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:39:42 UTC) from {username='slave'} at 127.0.0.1:36092
I20260810 14:39:42.140555 11667 tablet_replica.cc:330] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6: stopping tablet replica
I20260810 14:39:42.140782 11668 tablet_replica.cc:330] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: stopping tablet replica
I20260810 14:39:42.140916 11667 raft_consensus.cc:2229] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:39:42.141152 11668 raft_consensus.cc:2229] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:42.141371 11667 raft_consensus.cc:2258] T 5a595896d0d141809a593338195f7091 P 596a8e2300b549ba8285a11ac8615eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:42.141400 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11192
I20260810 14:39:42.141741 11665 tablet_replica.cc:330] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: stopping tablet replica
I20260810 14:39:42.142023 11665 raft_consensus.cc:2229] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:39:42.142294 11665 raft_consensus.cc:2258] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:42.143662 11668 raft_consensus.cc:2258] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:39:42.144414 11665 ts_tablet_manager.cc:1905] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:42.144924 11668 ts_tablet_manager.cc:1905] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:39:42.147045 11668 ts_tablet_manager.cc:1918] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:42.147182 11668 log.cc:1196] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-2/wal/wals/5a595896d0d141809a593338195f7091
I20260810 14:39:42.147543 11668 ts_tablet_manager.cc:1939] T 5a595896d0d141809a593338195f7091 P af8cc11f8f9a406482abbb605cb61382: Deleting consensus metadata
I20260810 14:39:42.148119 11665 ts_tablet_manager.cc:1918] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:39:42.148226 11665 log.cc:1196] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Deleting WAL directory at /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372517778458-21677-0/minicluster-data/ts-1/wal/wals/5a595896d0d141809a593338195f7091
I20260810 14:39:42.148551 11665 ts_tablet_manager.cc:1939] T 5a595896d0d141809a593338195f7091 P 51bfec0c018647bcbe4575d77987689b: Deleting consensus metadata
I20260810 14:39:42.148943 11112 catalog_manager.cc:4841] TS af8cc11f8f9a406482abbb605cb61382 (127.21.43.67:44343): tablet 5a595896d0d141809a593338195f7091 (table test-table [id=32bc053dff9548d592125fc9635e4914]) successfully deleted
I20260810 14:39:42.149770 11108 catalog_manager.cc:4841] TS 51bfec0c018647bcbe4575d77987689b (127.21.43.66:45153): tablet 5a595896d0d141809a593338195f7091 (table test-table [id=32bc053dff9548d592125fc9635e4914]) successfully deleted
W20260810 14:39:42.152762 11110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.43.65:38847 (error 108)
I20260810 14:39:42.153632 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11336
I20260810 14:39:42.153895 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:42.153569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.65:38847 (local address 127.0.0.1:58154)
0810 14:39:42.153695 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 14:39:42.153793 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.65:38847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:39:42.159675 11110 catalog_manager.cc:4584] TS 596a8e2300b549ba8285a11ac8615eb6 (127.21.43.65:38847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a595896d0d141809a593338195f7091: Network error: Client connection negotiation failed: client connection to 127.21.43.65:38847: connect: Connection refused (error 111)
I20260810 14:39:42.163563 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11479
I20260810 14:39:42.171502 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11091
2026-08-10T14:39:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1284 ms)
[----------] 6 tests from EncryptionPolicyTest (11591 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:42 dist-test-slave-chsk krb5kdc[11674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:42 dist-test-slave-chsk krb5kdc[11674](info): set up 2 sockets
Aug 10 14:39:42 dist-test-slave-chsk krb5kdc[11674](info): commencing operation
krb5kdc: starting...
W20260810 14:39:43.265842 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:43.393766 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:43.507371 11687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:43.507721 11687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:43.507812 11687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:43.512055 11687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:43.512164 11687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:43.512212 11687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:43.512274 11687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:43.512312 11687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:43.517827 11687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:43.519047 11687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:43.520236 11687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:43.529291 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:43.529251 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:43.529266 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:43.529757 11687 server_base.cc:1034] running on GCE node
I20260810 14:39:43.530409 11687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:43.531611 11687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:43.532805 11687 hybrid_clock.cc:648] HybridClock initialized: now 1786372783532775 us; error 46 us; skew 500 ppm
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:43.535660 11687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:43.536652 11687 webserver.cc:460] Webserver started at http://127.0.0.1:36685/ using document root <none> and password file <none>
I20260810 14:39:43.537252 11687 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:43.537349 11687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:43.537597 11687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:43.539820 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "dde72bd08a494bfbacdb90e0620bb8e2"
format_stamp: "Formatted at 2026-08-10 14:39:43 on dist-test-slave-chsk"
server_key: "bf9b182a8601c5da488b00438e486e34"
server_key_iv: "5de50cba1f5c4160aae90c0e15c98536"
server_key_version: "encryptionkey@0"
I20260810 14:39:43.540445 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "dde72bd08a494bfbacdb90e0620bb8e2"
format_stamp: "Formatted at 2026-08-10 14:39:43 on dist-test-slave-chsk"
server_key: "bf9b182a8601c5da488b00438e486e34"
server_key_iv: "5de50cba1f5c4160aae90c0e15c98536"
server_key_version: "encryptionkey@0"
I20260810 14:39:43.544464 11687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:39:43.546850 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:43.548169 11687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:39:43.548327 11687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "dde72bd08a494bfbacdb90e0620bb8e2"
format_stamp: "Formatted at 2026-08-10 14:39:43 on dist-test-slave-chsk"
server_key: "bf9b182a8601c5da488b00438e486e34"
server_key_iv: "5de50cba1f5c4160aae90c0e15c98536"
server_key_version: "encryptionkey@0"
I20260810 14:39:43.548445 11687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:43.566048 11687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:43.569069 11687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:43.569293 11687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:43.576993 11687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38437
I20260810 14:39:43.577065 11763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38437 every 8 connection(s)
I20260810 14:39:43.578032 11687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:43.580875 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11687
I20260810 14:39:43.580924 11764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:43.581036 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:43.581367 21677 external_mini_cluster.cc:1428] Setting key 95b13200ac2beff062a12a69a462441e
I20260810 14:39:43.588507 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: Bootstrap starting.
I20260810 14:39:43.591037 11764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: Neither blocks nor log segments found. Creating new log.
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:39:43.591929 11764 log.cc:824] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:43.594461 11764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: No bootstrap required, opened a new log
I20260810 14:39:43.597303 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:43.582867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37413 (local address 127.0.0.1:38437)
0810 14:39:43.583350 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:43.583369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:43.583413 (+ 44us) server_negotiation.cc:407] Connection header received
0810 14:39:43.584551 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:43.584578 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:43.585027 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:43.585403 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:43.586111 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:43.587678 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:43.588527 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:43.589189 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:43.589347 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:43.591953 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:43.591960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:43.591976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:43.592011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:43.594015 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:43.594552 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:43.594559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:43.594565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:43.594705 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:43.595035 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:43.595038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:43.595040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:43.595585 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:43.595792 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:43.596014 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 14:39:43.596245 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
I20260810 14:39:43.599042 11764 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260810 14:39:43.599385 11764 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:43.599493 11764 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde72bd08a494bfbacdb90e0620bb8e2, State: Initialized, Role: FOLLOWER
I20260810 14:39:43.599943 11764 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260810 14:39:43.600148 11764 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:43.600216 11764 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:43.600327 11764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:43.600598 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:43.601377 11764 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260810 14:39:43.601737 11764 leader_election.cc:302] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde72bd08a494bfbacdb90e0620bb8e2; no voters:
I20260810 14:39:43.602103 11764 leader_election.cc:288] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:43.602214 11773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:43.602440 11773 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [term 1 LEADER]: Becoming Leader. State: Replica: dde72bd08a494bfbacdb90e0620bb8e2, State: Running, Role: LEADER
I20260810 14:39:43.603336 11773 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } }
I20260810 14:39:43.603545 11764 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:43.605106 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde72bd08a494bfbacdb90e0620bb8e2. Latest consensus state: current_term: 1 leader_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260810 14:39:43.605244 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:43.605566 11774 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde72bd08a494bfbacdb90e0620bb8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38437 } } }
I20260810 14:39:43.607355 11774 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2 [sys.catalog]: This master's current role is: LEADER
W20260810 14:39:43.608666 11791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:39:43.608764 11791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:39:43.608877 11790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:43.609779 11790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:43.612686 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:43.602796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59177 (local address 127.0.0.1:38437)
0810 14:39:43.602914 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:43.602918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:43.603103 (+ 185us) server_negotiation.cc:407] Connection header received
0810 14:39:43.603338 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:43.603340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:43.603390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:43.603443 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:43.604402 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:43.605227 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:43.606582 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:43.606908 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:43.607021 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:43.608883 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:43.608887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:43.608890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:43.608918 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:43.610676 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:43.611296 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:43.611299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:43.611302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:43.611395 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:43.611797 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:43.611800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:43.611801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:43.611977 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:43.612135 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:43.612429 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 14:39:43.612544 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:39:43.616668 11790 catalog_manager.cc:1347] Generated new cluster ID: f6c1ee62fe5d4d1c8ba9246fd7b9e97b
I20260810 14:39:43.616739 11790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:43.631551 11790 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:43.632907 11790 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:43.642591 11790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dde72bd08a494bfbacdb90e0620bb8e2: Generated new TSK 0
I20260810 14:39:43.643419 11790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:43.654343 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11687
W20260810 14:39:43.664400 11672 connection.cc:535] client connection to 127.0.0.1:38437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38437): Connection reset by peer (error 104)
2026-08-10T14:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11799](info): set up 2 sockets
Aug 10 14:39:43 dist-test-slave-chsk krb5kdc[11799](info): commencing operation
krb5kdc: starting...
W20260810 14:39:44.754834 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:44 dist-test-slave-chsk krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:44.882783 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:44.996270 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:44.996592 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:44.996709 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:45.000886 11814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:45.000978 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:45.001017 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:45.001076 11814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:45.001114 11814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:45.006604 11814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:45.007861 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:45.009091 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:45.018644 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:45.018672 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:45.018698 11814 server_base.cc:1034] running on GCE node
W20260810 14:39:45.018783 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:45.019599 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:45.020694 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:45.021880 11814 hybrid_clock.cc:648] HybridClock initialized: now 1786372785021859 us; error 44 us; skew 500 ppm
Aug 10 14:39:45 dist-test-slave-chsk krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:45.024669 11814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:45.025576 11814 webserver.cc:460] Webserver started at http://127.0.0.1:44137/ using document root <none> and password file <none>
I20260810 14:39:45.026140 11814 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:45.026233 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:45.026470 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:45.028564 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "8e2f8fc2e89d4182ac25667caf240786"
format_stamp: "Formatted at 2026-08-10 14:39:45 on dist-test-slave-chsk"
server_key: "46906057d763adf05daef03811d7e6b2"
server_key_iv: "23013d59cae77990a8113104daacfd44"
server_key_version: "encryptionkey@0"
I20260810 14:39:45.029145 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "8e2f8fc2e89d4182ac25667caf240786"
format_stamp: "Formatted at 2026-08-10 14:39:45 on dist-test-slave-chsk"
server_key: "46906057d763adf05daef03811d7e6b2"
server_key_iv: "23013d59cae77990a8113104daacfd44"
server_key_version: "encryptionkey@0"
I20260810 14:39:45.032920 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:39:45.035343 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:45.036582 11814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:39:45.036759 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "8e2f8fc2e89d4182ac25667caf240786"
format_stamp: "Formatted at 2026-08-10 14:39:45 on dist-test-slave-chsk"
server_key: "46906057d763adf05daef03811d7e6b2"
server_key_iv: "23013d59cae77990a8113104daacfd44"
server_key_version: "encryptionkey@0"
I20260810 14:39:45.036880 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:45.059525 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:45.062681 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:45.062903 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:45.070590 11814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33659
I20260810 14:39:45.070619 11887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33659 every 8 connection(s)
I20260810 14:39:45.071609 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:45.074349 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:45.079916 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11814
I20260810 14:39:45.080072 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:45.080312 11888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786: Bootstrap starting.
I20260810 14:39:45.080381 21677 external_mini_cluster.cc:1428] Setting key 6cba4a7dfd4987da7784da123bfdcc98
I20260810 14:39:45.083189 11888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:45.084089 11888 log.cc:824] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:45.086301 11888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786: No bootstrap required, opened a new log
I20260810 14:39:45.090654 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:45.081794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41535 (local address 127.0.0.1:33659)
0810 14:39:45.082266 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:45.082287 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:45.082330 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:39:45.083248 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:45.083273 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:45.083718 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:45.084078 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:45.084541 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:45.085894 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:45.086812 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:45.087531 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:45.087701 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:45.088539 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:45.088545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:45.088562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:45.089080 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:45.089177 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:45.089215 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:45.089382 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:45.089515 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 14:39:45.089653 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260810 14:39:45.090868 11888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260810 14:39:45.091138 11888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:45.091236 11888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2f8fc2e89d4182ac25667caf240786, State: Initialized, Role: FOLLOWER
I20260810 14:39:45.091676 11888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260810 14:39:45.091853 11888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:45.091926 11888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:45.092022 11888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:45.093217 11888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260810 14:39:45.093571 11888 leader_election.cc:302] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2f8fc2e89d4182ac25667caf240786; no voters:
I20260810 14:39:45.093875 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:45.093909 11888 leader_election.cc:288] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:45.094012 11893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:45.094242 11893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2f8fc2e89d4182ac25667caf240786, State: Running, Role: LEADER
I20260810 14:39:45.094578 11893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } }
I20260810 14:39:45.095216 11888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:45.096379 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e2f8fc2e89d4182ac25667caf240786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } } }
I20260810 14:39:45.096524 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:45.096903 11895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e2f8fc2e89d4182ac25667caf240786. Latest consensus state: current_term: 1 leader_uuid: "8e2f8fc2e89d4182ac25667caf240786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2f8fc2e89d4182ac25667caf240786" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33659 } } }
I20260810 14:39:45.097012 11895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:45.097323 11908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:45.101169 11908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:45.101963 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:45.096308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55181 (local address 127.0.0.1:33659)
0810 14:39:45.096433 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:45.096437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:45.097615 (+ 1178us) server_negotiation.cc:407] Connection header received
0810 14:39:45.097881 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:45.097887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:45.097943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:45.098037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:45.098342 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:45.098854 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:45.099996 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:45.100348 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:45.100467 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:45.101428 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:45.101432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:45.101435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:45.101449 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:45.101479 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:45.101492 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:45.101595 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:45.101783 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:39:45.101824 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:39:45.107734 11908 catalog_manager.cc:1347] Generated new cluster ID: 17ec67620915490ebcea1e539a768bba
I20260810 14:39:45.107811 11908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:45.115751 11908 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:45.116993 11908 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:45.131031 11908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e2f8fc2e89d4182ac25667caf240786: Generated new TSK 0
I20260810 14:39:45.131947 11908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:45.141913 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11814
W20260810 14:39:45.151026 11796 connection.cc:535] client connection to 127.0.0.1:33659 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33659): Connection reset by peer (error 104)
2026-08-10T14:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:45 dist-test-slave-chsk krb5kdc[11922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:45 dist-test-slave-chsk krb5kdc[11922](info): set up 2 sockets
Aug 10 14:39:45 dist-test-slave-chsk krb5kdc[11922](info): commencing operation
krb5kdc: starting...
W20260810 14:39:46.254846 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:46 dist-test-slave-chsk krb5kdc[11922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:46.394532 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:46.516527 11938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:46.516885 11938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:46.516987 11938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:46.521281 11938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:46.521394 11938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:46.521433 11938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:46.521476 11938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:46.521515 11938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:46.528128 11938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:46.529480 11938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:46.530658 11938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:46.539969 11943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:46.539991 11946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:46.539985 11944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:46.540325 11938 server_base.cc:1034] running on GCE node
I20260810 14:39:46.541046 11938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:46.542383 11938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:46.543607 11938 hybrid_clock.cc:648] HybridClock initialized: now 1786372786543579 us; error 57 us; skew 500 ppm
Aug 10 14:39:46 dist-test-slave-chsk krb5kdc[11922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:46.558197 11938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:46.559487 11938 webserver.cc:460] Webserver started at http://127.0.0.1:39509/ using document root <none> and password file <none>
I20260810 14:39:46.560089 11938 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:46.560154 11938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:46.560338 11938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:46.562647 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc"
format_stamp: "Formatted at 2026-08-10 14:39:46 on dist-test-slave-chsk"
server_key: "f115fcafa4ee7be9c3a64f362e38160c"
server_key_iv: "b69129b68791e0155983943c505eab7c"
server_key_version: "encryptionkey@0"
I20260810 14:39:46.563308 11938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc"
format_stamp: "Formatted at 2026-08-10 14:39:46 on dist-test-slave-chsk"
server_key: "f115fcafa4ee7be9c3a64f362e38160c"
server_key_iv: "b69129b68791e0155983943c505eab7c"
server_key_version: "encryptionkey@0"
I20260810 14:39:46.567610 11938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:39:46.570369 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:46.571959 11938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:46.572166 11938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc"
format_stamp: "Formatted at 2026-08-10 14:39:46 on dist-test-slave-chsk"
server_key: "f115fcafa4ee7be9c3a64f362e38160c"
server_key_iv: "b69129b68791e0155983943c505eab7c"
server_key_version: "encryptionkey@0"
I20260810 14:39:46.572317 11938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:46.577497 11938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:46.581013 11938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:46.581264 11938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:46.589193 11938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35893
I20260810 14:39:46.589216 12013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35893 every 8 connection(s)
I20260810 14:39:46.590271 11938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:46.593245 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 11938
I20260810 14:39:46.593423 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:46.593498 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:46.593772 21677 external_mini_cluster.cc:1428] Setting key db3fd6858ec451c3e98c651c04123c26
I20260810 14:39:46.599812 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:46.595334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33601 (local address 127.0.0.1:35893)
0810 14:39:46.595796 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:46.595818 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:46.595867 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:39:46.596628 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:46.596654 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:46.597073 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:46.597492 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:46.598291 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:46.598297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:46.598313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:46.598326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:46.598395 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:46.598423 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:46.598569 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:46.598640 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 14:39:46.598801 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260810 14:39:46.601814 12014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc: Bootstrap starting.
I20260810 14:39:46.603001 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:46.604606 12014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:46.605500 12014 log.cc:824] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:46.607244 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:46.605335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39205 (local address 127.0.0.1:35893)
0810 14:39:46.605470 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:46.605475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:46.605657 (+ 182us) server_negotiation.cc:407] Connection header received
0810 14:39:46.605796 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:46.605799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:46.605844 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:46.605948 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:46.606822 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:46.606826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:46.606829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:46.606842 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:46.606871 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:46.606885 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:46.606976 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:46.607037 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 14:39:46.607076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:39:46.608234 12014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc: No bootstrap required, opened a new log
I20260810 14:39:46.612219 12014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260810 14:39:46.612524 12014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:46.612612 12014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1f3ed992e84c92a9edccec6cf9f5bc, State: Initialized, Role: FOLLOWER
I20260810 14:39:46.613106 12014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260810 14:39:46.613292 12014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:46.613379 12014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:46.613513 12014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:46.614609 12014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260810 14:39:46.615085 12014 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1f3ed992e84c92a9edccec6cf9f5bc; no voters:
I20260810 14:39:46.615537 12014 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:46.615629 12025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:46.615880 12025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [term 1 LEADER]: Becoming Leader. State: Replica: 4c1f3ed992e84c92a9edccec6cf9f5bc, State: Running, Role: LEADER
I20260810 14:39:46.616274 12025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260810 14:39:46.617141 12014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:46.618326 12027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1f3ed992e84c92a9edccec6cf9f5bc. Latest consensus state: current_term: 1 leader_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260810 14:39:46.618369 12026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1f3ed992e84c92a9edccec6cf9f5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260810 14:39:46.618577 12026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:46.618567 12027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:46.619069 12034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:46.622504 12034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:46.628672 12034 catalog_manager.cc:1347] Generated new cluster ID: fe59d6eddb0b4c78a07be204ff0147ca
I20260810 14:39:46.628778 12034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:46.641578 12034 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:46.642020 12034 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:46.649307 12034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1f3ed992e84c92a9edccec6cf9f5bc: Generated new TSK 0
I20260810 14:39:46.650339 12034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:46.663348 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 11938
2026-08-10T14:39:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:46 dist-test-slave-chsk krb5kdc[12047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:46 dist-test-slave-chsk krb5kdc[12047](info): set up 2 sockets
Aug 10 14:39:46 dist-test-slave-chsk krb5kdc[12047](info): commencing operation
krb5kdc: starting...
W20260810 14:39:47.777640 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:47 dist-test-slave-chsk krb5kdc[12047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:47.919240 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:48.041513 12063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:48.041807 12063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:48.041883 12063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:48.046079 12063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:48.046185 12063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:48.046226 12063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:48.046283 12063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:48.046320 12063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:48.052073 12063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:48.053260 12063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:48.054441 12063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:48.063956 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:48.064038 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:48.064157 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:48.064026 12063 server_base.cc:1034] running on GCE node
I20260810 14:39:48.064778 12063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:48.065966 12063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:48.067181 12063 hybrid_clock.cc:648] HybridClock initialized: now 1786372788067124 us; error 33 us; skew 500 ppm
Aug 10 14:39:48 dist-test-slave-chsk krb5kdc[12047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:48.070031 12063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:48.071100 12063 webserver.cc:460] Webserver started at http://127.0.0.1:41889/ using document root <none> and password file <none>
I20260810 14:39:48.071702 12063 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:48.071770 12063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:48.071946 12063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:48.074033 12063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "7fc55e7b31b74aa0875c2d38c4407f2a"
format_stamp: "Formatted at 2026-08-10 14:39:48 on dist-test-slave-chsk"
server_key: "7ec78031159a0d6c7bd11ed172f892cd"
server_key_iv: "c49db19668399980f9b5cf0ad53bce4a"
server_key_version: "encryptionkey@0"
I20260810 14:39:48.074571 12063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "7fc55e7b31b74aa0875c2d38c4407f2a"
format_stamp: "Formatted at 2026-08-10 14:39:48 on dist-test-slave-chsk"
server_key: "7ec78031159a0d6c7bd11ed172f892cd"
server_key_iv: "c49db19668399980f9b5cf0ad53bce4a"
server_key_version: "encryptionkey@0"
I20260810 14:39:48.078529 12063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 14:39:48.081174 12082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:48.082490 12063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:48.082650 12063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "7fc55e7b31b74aa0875c2d38c4407f2a"
format_stamp: "Formatted at 2026-08-10 14:39:48 on dist-test-slave-chsk"
server_key: "7ec78031159a0d6c7bd11ed172f892cd"
server_key_iv: "c49db19668399980f9b5cf0ad53bce4a"
server_key_version: "encryptionkey@0"
I20260810 14:39:48.082775 12063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:48.099838 12063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:48.103116 12063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:48.103387 12063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:48.110958 12063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46045
I20260810 14:39:48.110971 12141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46045 every 8 connection(s)
I20260810 14:39:48.111994 12063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:48.114845 12142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:48.117522 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12063
I20260810 14:39:48.117686 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:48.117992 21677 external_mini_cluster.cc:1428] Setting key 54edaa1b3fb0274651fb34fb58d2b8e7
I20260810 14:39:48.121681 12142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a: Bootstrap starting.
I20260810 14:39:48.124606 12142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:48.125532 12142 log.cc:824] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:48.127868 12142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a: No bootstrap required, opened a new log
Aug 10 14:39:48 dist-test-slave-chsk krb5kdc[12047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:39:48.132622 12142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260810 14:39:48.132938 12142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:48.133019 12142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc55e7b31b74aa0875c2d38c4407f2a, State: Initialized, Role: FOLLOWER
I20260810 14:39:48.133572 12142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260810 14:39:48.133764 12142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:48.133847 12142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:48.133965 12142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:48.135298 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:48.119557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43657 (local address 127.0.0.1:46045)
0810 14:39:48.120069 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:48.120091 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:48.120138 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:39:48.120985 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:48.121010 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:48.121448 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:48.121845 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:48.122689 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:48.124200 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:48.125114 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:48.125829 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:48.126015 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:48.128825 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:48.128831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:48.128847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:48.128881 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:48.131734 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:48.132196 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:48.132202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:48.132208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:48.132358 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:48.132644 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:48.132649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:48.132651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:48.133207 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:48.133423 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:48.133714 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 14:39:48.134226 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":130,"threads_started":1}
I20260810 14:39:48.135257 12142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260810 14:39:48.135699 12142 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc55e7b31b74aa0875c2d38c4407f2a; no voters:
I20260810 14:39:48.136101 12142 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:48.136258 12148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:48.136556 12148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [term 1 LEADER]: Becoming Leader. State: Replica: 7fc55e7b31b74aa0875c2d38c4407f2a, State: Running, Role: LEADER
I20260810 14:39:48.136929 12148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } }
I20260810 14:39:48.137495 12142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:48.138549 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:48.138774 12150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc55e7b31b74aa0875c2d38c4407f2a. Latest consensus state: current_term: 1 leader_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260810 14:39:48.138908 12150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:48.139288 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc55e7b31b74aa0875c2d38c4407f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46045 } } }
I20260810 14:39:48.139384 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:48.140584 12158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:48.144081 12158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:48.150523 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:48.141075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45855 (local address 127.0.0.1:46045)
0810 14:39:48.141191 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:48.141195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:48.141449 (+ 254us) server_negotiation.cc:407] Connection header received
0810 14:39:48.141588 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:48.141591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:48.141643 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:48.141710 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:48.142600 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:48.143452 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:48.144560 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:48.144916 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:48.145050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:48.146698 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:48.146701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:48.146703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:48.146729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:48.148256 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:48.148780 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:48.148784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:48.148786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:48.148884 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:48.149534 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:48.149539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:48.149541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:48.149780 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:48.149918 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:48.150209 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 14:39:48.150354 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:39:48.151120 12158 catalog_manager.cc:1347] Generated new cluster ID: 648b941ab4694c578d2b470882870810
I20260810 14:39:48.151223 12158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:48.172109 12158 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:48.173247 12158 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:48.178705 12158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc55e7b31b74aa0875c2d38c4407f2a: Generated new TSK 0
I20260810 14:39:48.179512 12158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:48.184973 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12063
2026-08-10T14:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:48 dist-test-slave-chsk krb5kdc[12175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:48 dist-test-slave-chsk krb5kdc[12175](info): set up 2 sockets
Aug 10 14:39:48 dist-test-slave-chsk krb5kdc[12175](info): commencing operation
krb5kdc: starting...
W20260810 14:39:49.291391 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:49 dist-test-slave-chsk krb5kdc[12175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:49.430075 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:49.544627 12188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:49.544939 12188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:49.545032 12188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:49.548983 12188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:49.549077 12188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:49.549120 12188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:49.549180 12188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:49.549212 12188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:49.554478 12188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:49.555707 12188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:49.556770 12188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:49.565497 12194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:49.565526 12197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:49.565505 12193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:49.565701 12188 server_base.cc:1034] running on GCE node
I20260810 14:39:49.566470 12188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:49.567668 12188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:49.568854 12188 hybrid_clock.cc:648] HybridClock initialized: now 1786372789568831 us; error 45 us; skew 500 ppm
Aug 10 14:39:49 dist-test-slave-chsk krb5kdc[12175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:49.571632 12188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:49.572430 12188 webserver.cc:460] Webserver started at http://127.0.0.1:43549/ using document root <none> and password file <none>
I20260810 14:39:49.572942 12188 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:49.573002 12188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:49.573164 12188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:49.575243 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "d33df4545c4941c680c6a1e70acf97e9"
format_stamp: "Formatted at 2026-08-10 14:39:49 on dist-test-slave-chsk"
server_key: "b0ff3fa173af193fa9db30fbec021e5d"
server_key_iv: "5f083dfc4ed6dfcb047549201e7e5cf6"
server_key_version: "encryptionkey@0"
I20260810 14:39:49.575764 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "d33df4545c4941c680c6a1e70acf97e9"
format_stamp: "Formatted at 2026-08-10 14:39:49 on dist-test-slave-chsk"
server_key: "b0ff3fa173af193fa9db30fbec021e5d"
server_key_iv: "5f083dfc4ed6dfcb047549201e7e5cf6"
server_key_version: "encryptionkey@0"
I20260810 14:39:49.579438 12188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 14:39:49.581794 12204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:49.583086 12188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:49.583299 12188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "d33df4545c4941c680c6a1e70acf97e9"
format_stamp: "Formatted at 2026-08-10 14:39:49 on dist-test-slave-chsk"
server_key: "b0ff3fa173af193fa9db30fbec021e5d"
server_key_iv: "5f083dfc4ed6dfcb047549201e7e5cf6"
server_key_version: "encryptionkey@0"
I20260810 14:39:49.583406 12188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:49.604784 12188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:49.608290 12188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:49.608525 12188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:49.615797 12188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37477
I20260810 14:39:49.615865 12264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37477 every 8 connection(s)
I20260810 14:39:49.616806 12188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:49.617441 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12188
I20260810 14:39:49.617625 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:49.617942 21677 external_mini_cluster.cc:1428] Setting key 9ad5158b5985331583f11ad1c6283477
I20260810 14:39:49.620026 12265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:49.626631 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9: Bootstrap starting.
I20260810 14:39:49.627341 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:49.619418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57905 (local address 127.0.0.1:37477)
0810 14:39:49.619870 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:49.619886 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:49.619922 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:39:49.620598 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:49.620614 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:49.620935 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:49.621214 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:49.621594 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:49.622610 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:49.623872 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:49.624360 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:49.624487 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:49.625427 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:49.625432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:49.625444 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:49.625885 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:49.625944 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:49.625967 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:49.626064 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:49.626218 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:39:49.626446 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":159,"threads_started":1}
I20260810 14:39:49.629001 12265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:49.629912 12265 log.cc:824] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:49.630262 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:49.632576 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9: No bootstrap required, opened a new log
I20260810 14:39:49.637437 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:49.633169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58677 (local address 127.0.0.1:37477)
0810 14:39:49.633308 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:49.633312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:49.633494 (+ 182us) server_negotiation.cc:407] Connection header received
0810 14:39:49.633634 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:49.633636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:49.633677 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:49.633744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:49.634057 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:49.634560 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:49.635791 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:49.636007 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:49.636090 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:49.636931 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:49.636935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:49.636937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:49.636948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:49.636972 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:49.636982 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:49.637050 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:49.637178 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 14:39:49.637235 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:39:49.637355 12265 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260810 14:39:49.637681 12265 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:49.637754 12265 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33df4545c4941c680c6a1e70acf97e9, State: Initialized, Role: FOLLOWER
I20260810 14:39:49.638250 12265 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260810 14:39:49.638422 12265 raft_consensus.cc:388] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:49.638499 12265 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:49.638602 12265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:49.639675 12265 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260810 14:39:49.640033 12265 leader_election.cc:302] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d33df4545c4941c680c6a1e70acf97e9; no voters:
I20260810 14:39:49.640406 12265 leader_election.cc:288] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:49.640533 12275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:49.640805 12275 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [term 1 LEADER]: Becoming Leader. State: Replica: d33df4545c4941c680c6a1e70acf97e9, State: Running, Role: LEADER
I20260810 14:39:49.641196 12275 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } }
I20260810 14:39:49.641745 12265 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:49.643023 12276 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d33df4545c4941c680c6a1e70acf97e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260810 14:39:49.643059 12277 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33df4545c4941c680c6a1e70acf97e9. Latest consensus state: current_term: 1 leader_uuid: "d33df4545c4941c680c6a1e70acf97e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33df4545c4941c680c6a1e70acf97e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37477 } } }
I20260810 14:39:49.643181 12276 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:49.643191 12277 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:49.643561 12285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:49.646914 12285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:49.652997 12285 catalog_manager.cc:1347] Generated new cluster ID: f2d1e896e08d4707ac9d8b9b08e5a37c
I20260810 14:39:49.653074 12285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:49.671454 12285 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:49.672610 12285 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:49.681177 12285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d33df4545c4941c680c6a1e70acf97e9: Generated new TSK 0
I20260810 14:39:49.681892 12285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:49.686836 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12188
2026-08-10T14:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:49 dist-test-slave-chsk krb5kdc[12298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:49 dist-test-slave-chsk krb5kdc[12298](info): set up 2 sockets
Aug 10 14:39:49 dist-test-slave-chsk krb5kdc[12298](info): commencing operation
krb5kdc: starting...
W20260810 14:39:50.786777 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:50 dist-test-slave-chsk krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:50.918191 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:51.031116 12312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:51.031487 12312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:51.031582 12312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:51.036856 12312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:51.036957 12312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:51.036994 12312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:51.037032 12312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:51.037066 12312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:51.043268 12312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:51.044711 12312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:51.045838 12312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:51.054827 12318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:51.054822 12321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:51.054881 12312 server_base.cc:1034] running on GCE node
W20260810 14:39:51.054819 12317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:51.055724 12312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:51.056870 12312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:51.058076 12312 hybrid_clock.cc:648] HybridClock initialized: now 1786372791058056 us; error 36 us; skew 500 ppm
Aug 10 14:39:51 dist-test-slave-chsk krb5kdc[12298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:51.061082 12312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:51.062077 12312 webserver.cc:460] Webserver started at http://127.0.0.1:35205/ using document root <none> and password file <none>
I20260810 14:39:51.062670 12312 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:51.062765 12312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:51.063011 12312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:51.065181 12312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "b117ce6cab884e92b909a28603c6dd18"
format_stamp: "Formatted at 2026-08-10 14:39:51 on dist-test-slave-chsk"
server_key: "5459cff62df60fce434c66a1140e43df"
server_key_iv: "9ac7b92ce9341d17df4ad11868bc364a"
server_key_version: "encryptionkey@0"
I20260810 14:39:51.065773 12312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "b117ce6cab884e92b909a28603c6dd18"
format_stamp: "Formatted at 2026-08-10 14:39:51 on dist-test-slave-chsk"
server_key: "5459cff62df60fce434c66a1140e43df"
server_key_iv: "9ac7b92ce9341d17df4ad11868bc364a"
server_key_version: "encryptionkey@0"
I20260810 14:39:51.069759 12312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 14:39:51.072320 12330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:51.073576 12312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:51.073725 12312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "b117ce6cab884e92b909a28603c6dd18"
format_stamp: "Formatted at 2026-08-10 14:39:51 on dist-test-slave-chsk"
server_key: "5459cff62df60fce434c66a1140e43df"
server_key_iv: "9ac7b92ce9341d17df4ad11868bc364a"
server_key_version: "encryptionkey@0"
I20260810 14:39:51.073845 12312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:51.078631 12312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:51.081905 12312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:51.082152 12312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:51.090075 12312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38345
I20260810 14:39:51.090099 12390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38345 every 8 connection(s)
I20260810 14:39:51.091092 12312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:51.094022 12391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:51.094851 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12312
I20260810 14:39:51.095010 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:51.095402 21677 external_mini_cluster.cc:1428] Setting key 7e73e5dc07dc25e469664c8b3e2469f5
I20260810 14:39:51.100684 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: Bootstrap starting.
I20260810 14:39:51.100834 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:51.096905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39909 (local address 127.0.0.1:38345)
0810 14:39:51.097387 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:51.097406 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:51.097443 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:39:51.098069 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:51.098085 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:51.098377 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:51.098720 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:51.099574 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:51.099582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:51.099599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:51.099610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:51.099665 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:51.099684 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:51.099848 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:51.099922 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:39:51.100090 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":149,"threads_started":1}
I20260810 14:39:51.103531 12391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:51.103852 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:51.104343 12391 log.cc:824] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:51.106813 12391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: No bootstrap required, opened a new log
I20260810 14:39:51.108050 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:51.106216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57669 (local address 127.0.0.1:38345)
0810 14:39:51.106358 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:51.106363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:51.106564 (+ 201us) server_negotiation.cc:407] Connection header received
0810 14:39:51.106694 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:51.106696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:51.106734 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:51.106800 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:51.107632 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:51.107635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:51.107637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:51.107648 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:51.107675 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:51.107686 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:51.107766 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:51.107862 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 14:39:51.107896 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:39:51.111084 12391 raft_consensus.cc:348] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } }
I20260810 14:39:51.111445 12391 raft_consensus.cc:374] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:51.111513 12391 raft_consensus.cc:729] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b117ce6cab884e92b909a28603c6dd18, State: Initialized, Role: FOLLOWER
I20260810 14:39:51.111954 12391 consensus_queue.cc:260] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } }
I20260810 14:39:51.112124 12391 raft_consensus.cc:388] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:51.112214 12391 raft_consensus.cc:482] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:51.112339 12391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:51.113405 12391 raft_consensus.cc:504] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } }
I20260810 14:39:51.113781 12391 leader_election.cc:302] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b117ce6cab884e92b909a28603c6dd18; no voters:
I20260810 14:39:51.114207 12391 leader_election.cc:288] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:51.114300 12401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:51.114513 12401 raft_consensus.cc:686] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [term 1 LEADER]: Becoming Leader. State: Replica: b117ce6cab884e92b909a28603c6dd18, State: Running, Role: LEADER
I20260810 14:39:51.114841 12401 consensus_queue.cc:237] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } }
I20260810 14:39:51.115715 12391 sys_catalog.cc:564] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:51.117084 12403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b117ce6cab884e92b909a28603c6dd18. Latest consensus state: current_term: 1 leader_uuid: "b117ce6cab884e92b909a28603c6dd18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } } }
I20260810 14:39:51.117062 12402 sys_catalog.cc:455] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b117ce6cab884e92b909a28603c6dd18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b117ce6cab884e92b909a28603c6dd18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38345 } } }
I20260810 14:39:51.117267 12402 sys_catalog.cc:458] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:51.117552 12403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18 [sys.catalog]: This master's current role is: LEADER
W20260810 14:39:51.120947 12419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:39:51.121064 12419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:39:51.121140 12412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:51.121932 12412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:51.128343 12412 catalog_manager.cc:1347] Generated new cluster ID: a4ed3b3b2edc4530bb97216ff510bab5
I20260810 14:39:51.128439 12412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:51.147640 12412 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:51.148013 12412 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:51.158584 12412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b117ce6cab884e92b909a28603c6dd18: Generated new TSK 0
I20260810 14:39:51.159447 12412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:51.172027 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12312
2026-08-10T14:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:51 dist-test-slave-chsk krb5kdc[12424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:51 dist-test-slave-chsk krb5kdc[12424](info): set up 2 sockets
Aug 10 14:39:51 dist-test-slave-chsk krb5kdc[12424](info): commencing operation
krb5kdc: starting...
W20260810 14:39:52.278471 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:52.411026 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:52.522782 12439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:52.523039 12439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:52.523080 12439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:52.527134 12439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:52.527208 12439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:52.527227 12439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:52.527271 12439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:52.527305 12439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:52.532517 12439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:52.533581 12439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:52.534632 12439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:52.543424 12444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:52.543438 12449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:52.543433 12447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:52.543625 12439 server_base.cc:1034] running on GCE node
I20260810 14:39:52.544196 12439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:52.545238 12439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:52.546432 12439 hybrid_clock.cc:648] HybridClock initialized: now 1786372792546409 us; error 49 us; skew 500 ppm
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:52.553681 12439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:52.554647 12439 webserver.cc:460] Webserver started at http://127.0.0.1:46059/ using document root <none> and password file <none>
I20260810 14:39:52.555285 12439 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:52.555380 12439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:52.555625 12439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:52.557761 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb"
format_stamp: "Formatted at 2026-08-10 14:39:52 on dist-test-slave-chsk"
server_key: "812d0e0f0a45cc8bba71cceaecb7038d"
server_key_iv: "3f9430d4ca996a661d6ce595f1c232ec"
server_key_version: "encryptionkey@0"
I20260810 14:39:52.558324 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb"
format_stamp: "Formatted at 2026-08-10 14:39:52 on dist-test-slave-chsk"
server_key: "812d0e0f0a45cc8bba71cceaecb7038d"
server_key_iv: "3f9430d4ca996a661d6ce595f1c232ec"
server_key_version: "encryptionkey@0"
I20260810 14:39:52.562121 12439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:39:52.564522 12456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:52.565693 12439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:39:52.565847 12439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb"
format_stamp: "Formatted at 2026-08-10 14:39:52 on dist-test-slave-chsk"
server_key: "812d0e0f0a45cc8bba71cceaecb7038d"
server_key_iv: "3f9430d4ca996a661d6ce595f1c232ec"
server_key_version: "encryptionkey@0"
I20260810 14:39:52.565963 12439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:52.583598 12439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:52.586462 12439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:52.586686 12439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:52.594148 12439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34273
I20260810 14:39:52.594177 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34273 every 8 connection(s)
I20260810 14:39:52.595225 12439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:52.598011 12520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:52.598062 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12439
I20260810 14:39:52.598223 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:52.598541 21677 external_mini_cluster.cc:1428] Setting key ab072425206fe6a1905be6c0c69d29a7
I20260810 14:39:52.605118 12520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb: Bootstrap starting.
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:39:52.608047 12520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:52.608800 12520 log.cc:824] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:52.610793 12520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb: No bootstrap required, opened a new log
I20260810 14:39:52.613358 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:52.600047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:36165 (local address 127.0.0.1:34273)
0810 14:39:52.600480 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:52.600504 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:52.600539 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:39:52.601271 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:52.601287 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:52.601621 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:52.601891 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:52.602869 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:52.603918 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:52.605071 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:52.605581 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:52.605699 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:52.608591 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:52.608597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:52.608610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:52.608638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:52.610393 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:52.610925 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:52.610935 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:52.610940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:52.611049 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:52.611432 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:52.611436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:52.611437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:52.611864 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:52.612088 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:52.612425 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 14:39:52.612648 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260810 14:39:52.614755 12520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } }
I20260810 14:39:52.615041 12520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:52.615123 12520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e36a177c0ce4ff18b8f3dda580a2bfb, State: Initialized, Role: FOLLOWER
I20260810 14:39:52.615607 12520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } }
I20260810 14:39:52.615736 12520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:52.615824 12520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:52.615962 12520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:52.616482 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:52.617142 12520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } }
I20260810 14:39:52.617528 12520 leader_election.cc:302] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e36a177c0ce4ff18b8f3dda580a2bfb; no voters:
I20260810 14:39:52.617899 12520 leader_election.cc:288] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:52.618002 12526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:52.618273 12526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [term 1 LEADER]: Becoming Leader. State: Replica: 9e36a177c0ce4ff18b8f3dda580a2bfb, State: Running, Role: LEADER
I20260810 14:39:52.618670 12526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } }
I20260810 14:39:52.619342 12520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:52.620042 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e36a177c0ce4ff18b8f3dda580a2bfb. Latest consensus state: current_term: 1 leader_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } } }
I20260810 14:39:52.620218 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:52.620579 12530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e36a177c0ce4ff18b8f3dda580a2bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34273 } } }
I20260810 14:39:52.620750 12530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:52.620653 12540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:52.624245 12540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:52.630220 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:52.618852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:51403 (local address 127.0.0.1:34273)
0810 14:39:52.618956 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:52.618960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:52.620967 (+ 2007us) server_negotiation.cc:407] Connection header received
0810 14:39:52.621156 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:52.621159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:52.621208 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:52.621261 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:52.622249 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:52.623043 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:52.624231 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:52.624554 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:52.624673 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:52.626438 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:52.626442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:52.626445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:52.626472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:52.628241 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:52.628764 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:52.628768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:52.628770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:52.628884 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:52.629291 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:52.629294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:52.629295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:52.629505 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:52.629660 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:52.629969 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 14:39:52.630070 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:39:52.630741 12540 catalog_manager.cc:1347] Generated new cluster ID: d77818e77ba0473b96a79213a66038ac
I20260810 14:39:52.630861 12540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:52.643126 12540 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:52.644177 12540 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:52.649077 12540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e36a177c0ce4ff18b8f3dda580a2bfb: Generated new TSK 0
I20260810 14:39:52.649767 12540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:52.654662 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12439
W20260810 14:39:52.662916 12422 connection.cc:535] client connection to 127.0.0.1:34273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34273): Connection reset by peer (error 104)
2026-08-10T14:39:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12554](info): set up 2 sockets
Aug 10 14:39:52 dist-test-slave-chsk krb5kdc[12554](info): commencing operation
krb5kdc: starting...
W20260810 14:39:53.750155 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:53 dist-test-slave-chsk krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:53.881872 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:53.997473 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:53.997815 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:53.997901 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:54.001972 12570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:54.002106 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:54.002144 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:54.002183 12570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:54.002223 12570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:54.007800 12570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:54.008981 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:54.010066 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:54.019588 12576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:54.019582 12579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:54.019656 12570 server_base.cc:1034] running on GCE node
W20260810 14:39:54.019582 12577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:54.020521 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:54.021745 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:54.022984 12570 hybrid_clock.cc:648] HybridClock initialized: now 1786372794022956 us; error 41 us; skew 500 ppm
Aug 10 14:39:54 dist-test-slave-chsk krb5kdc[12554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:54.026139 12570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:54.027092 12570 webserver.cc:460] Webserver started at http://127.0.0.1:45215/ using document root <none> and password file <none>
I20260810 14:39:54.027705 12570 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:54.027773 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:54.027946 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:54.030030 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "44a4693af2d84762a65ee69fc77b537b"
format_stamp: "Formatted at 2026-08-10 14:39:54 on dist-test-slave-chsk"
server_key: "8ded60f57e94ece18c3e4be919fd62e2"
server_key_iv: "ed5a24f1a992a65c108192dbcf8f6c4b"
server_key_version: "encryptionkey@0"
I20260810 14:39:54.030597 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "44a4693af2d84762a65ee69fc77b537b"
format_stamp: "Formatted at 2026-08-10 14:39:54 on dist-test-slave-chsk"
server_key: "8ded60f57e94ece18c3e4be919fd62e2"
server_key_iv: "ed5a24f1a992a65c108192dbcf8f6c4b"
server_key_version: "encryptionkey@0"
I20260810 14:39:54.034435 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:39:54.036921 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:54.038224 12570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:39:54.038335 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "44a4693af2d84762a65ee69fc77b537b"
format_stamp: "Formatted at 2026-08-10 14:39:54 on dist-test-slave-chsk"
server_key: "8ded60f57e94ece18c3e4be919fd62e2"
server_key_iv: "ed5a24f1a992a65c108192dbcf8f6c4b"
server_key_version: "encryptionkey@0"
I20260810 14:39:54.038477 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:54.050912 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:54.054179 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:54.054354 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:54.061887 12570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36811
I20260810 14:39:54.061959 12645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36811 every 8 connection(s)
I20260810 14:39:54.062973 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:54.066135 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:54.069428 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12570
I20260810 14:39:54.069595 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:54.069872 21677 external_mini_cluster.cc:1428] Setting key a7c74adf54bec6cba61461c333d748c8
I20260810 14:39:54.072960 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b: Bootstrap starting.
I20260810 14:39:54.075604 12646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:54.076452 12646 log.cc:824] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:54.079183 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b: No bootstrap required, opened a new log
I20260810 14:39:54.080163 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:54.071484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:60495 (local address 127.0.0.1:36811)
0810 14:39:54.072012 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:54.072028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:54.072075 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:39:54.072719 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:54.072737 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:54.073162 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:54.073537 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:54.073977 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:54.074995 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:54.076140 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:54.076698 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:54.076886 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:54.077929 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:54.077936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:54.077955 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:54.078493 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:54.078570 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:54.078604 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:54.078748 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:54.078960 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:39:54.079122 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":159,"threads_started":1}
I20260810 14:39:54.083626 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:54.083846 12646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } }
I20260810 14:39:54.084123 12646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:54.084218 12646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a4693af2d84762a65ee69fc77b537b, State: Initialized, Role: FOLLOWER
I20260810 14:39:54.084725 12646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } }
I20260810 14:39:54.084890 12646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:54.084982 12646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:54.085116 12646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:54.086468 12646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } }
I20260810 14:39:54.086903 12646 leader_election.cc:302] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44a4693af2d84762a65ee69fc77b537b; no voters:
I20260810 14:39:54.087905 12646 leader_election.cc:288] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:54.087975 12657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:54.088227 12657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [term 1 LEADER]: Becoming Leader. State: Replica: 44a4693af2d84762a65ee69fc77b537b, State: Running, Role: LEADER
I20260810 14:39:54.088609 12657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } }
I20260810 14:39:54.089377 12646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:54.089998 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a4693af2d84762a65ee69fc77b537b. Latest consensus state: current_term: 1 leader_uuid: "44a4693af2d84762a65ee69fc77b537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } } }
I20260810 14:39:54.090222 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:54.090477 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44a4693af2d84762a65ee69fc77b537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a4693af2d84762a65ee69fc77b537b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36811 } } }
I20260810 14:39:54.090744 12666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:54.090925 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:54.091526 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:54.086039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:37291 (local address 127.0.0.1:36811)
0810 14:39:54.086186 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:54.086191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:54.086363 (+ 172us) server_negotiation.cc:407] Connection header received
0810 14:39:54.086454 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:54.086457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:54.086516 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:54.086610 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:54.086939 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:54.087680 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:54.088834 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:54.089101 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:54.089235 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:54.090920 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:54.090924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:54.090927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:54.090943 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:54.090977 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:54.090991 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:54.091127 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:54.091352 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 14:39:54.091394 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:39:54.094496 12666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:54.100353 12666 catalog_manager.cc:1347] Generated new cluster ID: 23fdd1bf0bdf4e7f8c10087d28d03d80
I20260810 14:39:54.100440 12666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:54.113183 12666 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:54.114516 12666 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:54.129012 12666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44a4693af2d84762a65ee69fc77b537b: Generated new TSK 0
I20260810 14:39:54.129940 12666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:54.142787 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12570
W20260810 14:39:54.152621 12551 connection.cc:535] client connection to 127.0.0.1:36811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36811): Connection reset by peer (error 104)
2026-08-10T14:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:54 dist-test-slave-chsk krb5kdc[12685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:54 dist-test-slave-chsk krb5kdc[12685](info): set up 2 sockets
Aug 10 14:39:54 dist-test-slave-chsk krb5kdc[12685](info): commencing operation
krb5kdc: starting...
W20260810 14:39:55.246549 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:55 dist-test-slave-chsk krb5kdc[12685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:55.386196 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:55.518486 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:55.519026 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:55.519126 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:55.526970 12701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:55.527094 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:55.527135 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:55.527195 12701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:55.527218 12701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:55.535944 12701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:55.537295 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:55.538790 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:55.548065 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:55.548362 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:55.548062 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:55.548729 12701 server_base.cc:1034] running on GCE node
I20260810 14:39:55.549431 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:55.550724 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:55.551973 12701 hybrid_clock.cc:648] HybridClock initialized: now 1786372795551944 us; error 42 us; skew 500 ppm
Aug 10 14:39:55 dist-test-slave-chsk krb5kdc[12685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:55.555264 12701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:55.556308 12701 webserver.cc:460] Webserver started at http://127.0.0.1:42967/ using document root <none> and password file <none>
I20260810 14:39:55.556917 12701 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:55.557013 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:55.557279 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:55.559636 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "73ce01e911e547f48f2d918ec8328fc9"
format_stamp: "Formatted at 2026-08-10 14:39:55 on dist-test-slave-chsk"
server_key: "0507e3f9c73a8ff234290ff83694aa4b"
server_key_iv: "687bd6f87f7df77ec2da500c39de8429"
server_key_version: "encryptionkey@0"
I20260810 14:39:55.560307 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "73ce01e911e547f48f2d918ec8328fc9"
format_stamp: "Formatted at 2026-08-10 14:39:55 on dist-test-slave-chsk"
server_key: "0507e3f9c73a8ff234290ff83694aa4b"
server_key_iv: "687bd6f87f7df77ec2da500c39de8429"
server_key_version: "encryptionkey@0"
I20260810 14:39:55.564904 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:39:55.567955 12718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:55.569417 12701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:55.569594 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "73ce01e911e547f48f2d918ec8328fc9"
format_stamp: "Formatted at 2026-08-10 14:39:55 on dist-test-slave-chsk"
server_key: "0507e3f9c73a8ff234290ff83694aa4b"
server_key_iv: "687bd6f87f7df77ec2da500c39de8429"
server_key_version: "encryptionkey@0"
I20260810 14:39:55.569748 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:55.575057 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:55.578524 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:55.578744 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:55.586607 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42917
I20260810 14:39:55.586638 12772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42917 every 8 connection(s)
I20260810 14:39:55.587870 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:55.591480 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:55.595589 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12701
I20260810 14:39:55.595777 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:55.596086 21677 external_mini_cluster.cc:1428] Setting key 2f2dc9d3ed10a5d81e0325d21cbe8061
I20260810 14:39:55.598292 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9: Bootstrap starting.
I20260810 14:39:55.601502 12773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:55.602558 12773 log.cc:824] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:55.603312 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:55.598202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:36269 (local address 127.0.0.1:42917)
0810 14:39:55.598754 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:55.598774 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:55.598812 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:39:55.599896 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:55.599928 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:55.600328 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:55.600648 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:55.601704 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:55.601709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:55.601722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:55.601735 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:55.601808 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:55.601836 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:55.601942 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:55.602177 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:39:55.602326 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":221,"threads_started":1}
I20260810 14:39:55.604872 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9: No bootstrap required, opened a new log
I20260810 14:39:55.606458 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:55.610082 12773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260810 14:39:55.610351 12773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:55.610423 12773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce01e911e547f48f2d918ec8328fc9, State: Initialized, Role: FOLLOWER
I20260810 14:39:55.610984 12773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260810 14:39:55.611097 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:55.609136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:36209 (local address 127.0.0.1:42917)
0810 14:39:55.609273 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:55.609278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:55.609451 (+ 173us) server_negotiation.cc:407] Connection header received
0810 14:39:55.609569 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:55.609573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:55.609617 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:55.609711 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:55.610624 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:55.610628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:55.610631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:55.610645 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:55.610679 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:55.610692 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:55.610792 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:55.610883 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:39:55.610926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:39:55.611222 12773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:55.611286 12773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:55.611395 12773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:55.612639 12773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260810 14:39:55.612977 12773 leader_election.cc:302] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73ce01e911e547f48f2d918ec8328fc9; no voters:
I20260810 14:39:55.613266 12773 leader_election.cc:288] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:55.613425 12785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:55.613684 12785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 73ce01e911e547f48f2d918ec8328fc9, State: Running, Role: LEADER
I20260810 14:39:55.614086 12785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260810 14:39:55.614693 12773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:55.616009 12787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73ce01e911e547f48f2d918ec8328fc9. Latest consensus state: current_term: 1 leader_uuid: "73ce01e911e547f48f2d918ec8328fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } } }
I20260810 14:39:55.616006 12786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73ce01e911e547f48f2d918ec8328fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73ce01e911e547f48f2d918ec8328fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } } }
I20260810 14:39:55.616173 12786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:55.616175 12787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:55.616724 12797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:55.620352 12797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:55.627282 12797 catalog_manager.cc:1347] Generated new cluster ID: 6b9489de63224128b8057aae59e6e510
I20260810 14:39:55.627386 12797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:55.642969 12797 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:55.643407 12797 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:55.658979 12797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73ce01e911e547f48f2d918ec8328fc9: Generated new TSK 0
I20260810 14:39:55.659931 12797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:55.672092 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12701
2026-08-10T14:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:55 dist-test-slave-chsk krb5kdc[12810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:55 dist-test-slave-chsk krb5kdc[12810](info): set up 2 sockets
Aug 10 14:39:55 dist-test-slave-chsk krb5kdc[12810](info): commencing operation
krb5kdc: starting...
W20260810 14:39:56.778002 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:56 dist-test-slave-chsk krb5kdc[12810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:56.914021 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:57.031821 12824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:57.032148 12824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:57.032243 12824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:57.036271 12824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:57.036369 12824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:57.036409 12824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:57.036448 12824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:57.036479 12824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:57.041834 12824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:57.042969 12824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:57.044037 12824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:57.052775 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:57.052784 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:57.052784 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:57.053023 12824 server_base.cc:1034] running on GCE node
I20260810 14:39:57.053762 12824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:57.054972 12824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:57.056171 12824 hybrid_clock.cc:648] HybridClock initialized: now 1786372797056140 us; error 59 us; skew 500 ppm
Aug 10 14:39:57 dist-test-slave-chsk krb5kdc[12810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:57.059924 12824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:57.061043 12824 webserver.cc:460] Webserver started at http://127.0.0.1:45537/ using document root <none> and password file <none>
I20260810 14:39:57.061661 12824 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:57.061755 12824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:57.061993 12824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:57.064304 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "ddf383ef946647e1974c0c7994d237a7"
format_stamp: "Formatted at 2026-08-10 14:39:57 on dist-test-slave-chsk"
server_key: "dd47756ae329919557cc77550cbd3b86"
server_key_iv: "c4083e9a716abd8ace0bb8f0d404bdaa"
server_key_version: "encryptionkey@0"
I20260810 14:39:57.064908 12824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "ddf383ef946647e1974c0c7994d237a7"
format_stamp: "Formatted at 2026-08-10 14:39:57 on dist-test-slave-chsk"
server_key: "dd47756ae329919557cc77550cbd3b86"
server_key_iv: "c4083e9a716abd8ace0bb8f0d404bdaa"
server_key_version: "encryptionkey@0"
I20260810 14:39:57.069060 12824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 14:39:57.071918 12840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:57.073313 12824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:39:57.073501 12824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "ddf383ef946647e1974c0c7994d237a7"
format_stamp: "Formatted at 2026-08-10 14:39:57 on dist-test-slave-chsk"
server_key: "dd47756ae329919557cc77550cbd3b86"
server_key_iv: "c4083e9a716abd8ace0bb8f0d404bdaa"
server_key_version: "encryptionkey@0"
I20260810 14:39:57.073640 12824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:57.105952 12824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:57.109623 12824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:57.109856 12824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:57.117904 12824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39713
I20260810 14:39:57.117925 12897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39713 every 8 connection(s)
I20260810 14:39:57.118968 12824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:57.121869 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:57.122270 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12824
I20260810 14:39:57.122424 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:57.122740 21677 external_mini_cluster.cc:1428] Setting key f76d5f40c903bbbf7de65d7f269711ac
I20260810 14:39:57.128736 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7: Bootstrap starting.
I20260810 14:39:57.131377 12898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7: Neither blocks nor log segments found. Creating new log.
Aug 10 14:39:57 dist-test-slave-chsk krb5kdc[12810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:39:57.132274 12898 log.cc:824] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:57.134199 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7: No bootstrap required, opened a new log
I20260810 14:39:57.137290 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:57.124261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:60765 (local address 127.0.0.1:39713)
0810 14:39:57.124730 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:57.124745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:57.124782 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:39:57.125464 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:57.125480 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:57.125827 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:57.126171 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:57.126982 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:57.128037 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:57.128928 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:57.129596 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:57.129785 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:57.132575 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:57.132580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:57.132593 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:57.132620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:57.134412 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:57.134922 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:57.134932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:57.134936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:57.135039 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:57.135420 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:57.135424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:57.135425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:57.135808 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:57.136027 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:57.136369 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 14:39:57.136586 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
I20260810 14:39:57.137935 12898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } }
I20260810 14:39:57.138223 12898 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:57.138312 12898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf383ef946647e1974c0c7994d237a7, State: Initialized, Role: FOLLOWER
I20260810 14:39:57.138803 12898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } }
I20260810 14:39:57.138980 12898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:57.139050 12898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:57.139179 12898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:57.140396 12898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } }
I20260810 14:39:57.140445 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:57.140770 12898 leader_election.cc:302] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf383ef946647e1974c0c7994d237a7; no voters:
I20260810 14:39:57.141110 12898 leader_election.cc:288] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:57.141211 12904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:57.141394 12904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [term 1 LEADER]: Becoming Leader. State: Replica: ddf383ef946647e1974c0c7994d237a7, State: Running, Role: LEADER
I20260810 14:39:57.141757 12904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } }
I20260810 14:39:57.142467 12898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:57.143566 12906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf383ef946647e1974c0c7994d237a7. Latest consensus state: current_term: 1 leader_uuid: "ddf383ef946647e1974c0c7994d237a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } } }
I20260810 14:39:57.143711 12906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:57.144070 12905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf383ef946647e1974c0c7994d237a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf383ef946647e1974c0c7994d237a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39713 } } }
I20260810 14:39:57.144150 12918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:57.144173 12905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:57.147722 12918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:57.154014 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:57.142602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:53821 (local address 127.0.0.1:39713)
0810 14:39:57.142783 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:57.142787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:57.145003 (+ 2216us) server_negotiation.cc:407] Connection header received
0810 14:39:57.145137 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:57.145140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:57.145191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:57.145273 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:57.146161 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:57.146916 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:57.148064 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:57.148380 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:57.148460 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:57.150399 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:57.150403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:57.150406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:39:57.150441 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:57.151998 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:57.152508 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:57.152511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:57.152513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:57.152597 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:39:57.152983 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:39:57.152987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:39:57.152989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:39:57.153214 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:57.153371 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:57.153731 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 14:39:57.153843 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 14:39:57.154524 12918 catalog_manager.cc:1347] Generated new cluster ID: d36d138be0894af88714eee341e58ffa
I20260810 14:39:57.154585 12918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:57.163472 12918 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:57.164474 12918 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:57.173544 12918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf383ef946647e1974c0c7994d237a7: Generated new TSK 0
I20260810 14:39:57.174430 12918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:57.184679 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12824
W20260810 14:39:57.193507 12807 connection.cc:535] client connection to 127.0.0.1:39713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39713): Connection reset by peer (error 104)
2026-08-10T14:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1510 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:57 dist-test-slave-chsk krb5kdc[12930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:57 dist-test-slave-chsk krb5kdc[12930](info): set up 2 sockets
Aug 10 14:39:57 dist-test-slave-chsk krb5kdc[12930](info): commencing operation
krb5kdc: starting...
W20260810 14:39:58.294301 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:58 dist-test-slave-chsk krb5kdc[12930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:58.431293 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:39:58.545423 12945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:39:58.545733 12945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:39:58.545805 12945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:39:58.549834 12945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:39:58.549925 12945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:39:58.549965 12945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:39:58.550024 12945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:39:58.550071 12945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:39:58.555416 12945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:39:58.556473 12945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:39:58.557505 12945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:39:58.565944 12954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:58.565940 12951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:39:58.565941 12952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:39:58.565992 12945 server_base.cc:1034] running on GCE node
I20260810 14:39:58.566756 12945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:39:58.567907 12945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:39:58.569109 12945 hybrid_clock.cc:648] HybridClock initialized: now 1786372798569079 us; error 45 us; skew 500 ppm
Aug 10 14:39:58 dist-test-slave-chsk krb5kdc[12930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:39:58.577787 12945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:39:58.578774 12945 webserver.cc:460] Webserver started at http://127.0.0.1:36381/ using document root <none> and password file <none>
I20260810 14:39:58.579437 12945 fs_manager.cc:359] Metadata directory not provided
I20260810 14:39:58.579530 12945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:39:58.579777 12945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:39:58.581854 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "6a6ce2fe90fe407ea12bd366f728d086"
format_stamp: "Formatted at 2026-08-10 14:39:58 on dist-test-slave-chsk"
server_key: "727945d6df7afd869b048942302597a4"
server_key_iv: "820ada49fe76714bdd284199d4f7ae19"
server_key_version: "encryptionkey@0"
I20260810 14:39:58.582459 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "6a6ce2fe90fe407ea12bd366f728d086"
format_stamp: "Formatted at 2026-08-10 14:39:58 on dist-test-slave-chsk"
server_key: "727945d6df7afd869b048942302597a4"
server_key_iv: "820ada49fe76714bdd284199d4f7ae19"
server_key_version: "encryptionkey@0"
I20260810 14:39:58.586220 12945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 14:39:58.588703 12962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:39:58.589924 12945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 14:39:58.590082 12945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "6a6ce2fe90fe407ea12bd366f728d086"
format_stamp: "Formatted at 2026-08-10 14:39:58 on dist-test-slave-chsk"
server_key: "727945d6df7afd869b048942302597a4"
server_key_iv: "820ada49fe76714bdd284199d4f7ae19"
server_key_version: "encryptionkey@0"
I20260810 14:39:58.590245 12945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:39:58.604861 12945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:39:58.607849 12945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:39:58.608012 12945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:39:58.615083 12945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36709
I20260810 14:39:58.615104 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36709 every 8 connection(s)
I20260810 14:39:58.616092 12945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:39:58.618863 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:39:58.619179 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 12945
I20260810 14:39:58.619323 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:39:58.619659 21677 external_mini_cluster.cc:1428] Setting key 58536ffcf550d7acb12ea3681a0fbd8e
I20260810 14:39:58.625938 13019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086: Bootstrap starting.
I20260810 14:39:58.628206 13019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086: Neither blocks nor log segments found. Creating new log.
I20260810 14:39:58.628799 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:58.621147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:52037 (local address 127.0.0.1:36709)
0810 14:39:58.621554 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:58.621569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:58.621604 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:39:58.622393 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:58.622411 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:58.622743 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:58.623010 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:58.623392 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:58.624349 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:58.625544 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:58.625986 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:58.626169 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:58.627107 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:58.627112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:58.627123 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:58.627555 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:58.627611 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:58.627634 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:58.627773 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:58.627923 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 14:39:58.628060 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260810 14:39:58.628984 13019 log.cc:824] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086: Log is configured to *not* fsync() on all Append() calls
I20260810 14:39:58.631654 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:39:58.631835 13019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086: No bootstrap required, opened a new log
I20260810 14:39:58.636094 13019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260810 14:39:58.636353 13019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:39:58.636417 13019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6ce2fe90fe407ea12bd366f728d086, State: Initialized, Role: FOLLOWER
I20260810 14:39:58.636878 13019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260810 14:39:58.637001 13019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:39:58.637087 13019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:39:58.637197 13019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:39:58.638183 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:39:58.633986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:57973 (local address 127.0.0.1:36709)
0810 14:39:58.634101 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:39:58.634105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:39:58.634318 (+ 213us) server_negotiation.cc:407] Connection header received
0810 14:39:58.634468 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:39:58.634471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:39:58.634521 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:39:58.634602 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:39:58.634875 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:58.635462 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:58.636371 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:39:58.636680 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:39:58.636789 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:39:58.637717 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:39:58.637721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:39:58.637724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:39:58.637737 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:39:58.637769 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:39:58.637779 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:39:58.637876 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:39:58.637997 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:39:58.638031 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:39:58.638152 13019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260810 14:39:58.638549 13019 leader_election.cc:302] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a6ce2fe90fe407ea12bd366f728d086; no voters:
I20260810 14:39:58.638914 13019 leader_election.cc:288] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:39:58.639036 13030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:39:58.639310 13030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [term 1 LEADER]: Becoming Leader. State: Replica: 6a6ce2fe90fe407ea12bd366f728d086, State: Running, Role: LEADER
I20260810 14:39:58.639710 13030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } }
I20260810 14:39:58.640254 13019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:39:58.641453 13032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a6ce2fe90fe407ea12bd366f728d086. Latest consensus state: current_term: 1 leader_uuid: "6a6ce2fe90fe407ea12bd366f728d086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } } }
I20260810 14:39:58.641592 13032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:58.641907 13031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a6ce2fe90fe407ea12bd366f728d086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6ce2fe90fe407ea12bd366f728d086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36709 } } }
I20260810 14:39:58.642086 13040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:39:58.642098 13031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086 [sys.catalog]: This master's current role is: LEADER
I20260810 14:39:58.645186 13040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:39:58.651239 13040 catalog_manager.cc:1347] Generated new cluster ID: 9a24b53e3e2a4714bc81d6cbe1a95d87
I20260810 14:39:58.651319 13040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:39:58.659763 13040 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:39:58.660640 13040 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:39:58.667101 13040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a6ce2fe90fe407ea12bd366f728d086: Generated new TSK 0
I20260810 14:39:58.667794 13040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:39:58.675902 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 12945
W20260810 14:39:58.684945 12928 connection.cc:535] client connection to 127.0.0.1:36709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36709): Connection reset by peer (error 104)
2026-08-10T14:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:39:58 dist-test-slave-chsk krb5kdc[13053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:39:58 dist-test-slave-chsk krb5kdc[13053](info): set up 2 sockets
Aug 10 14:39:58 dist-test-slave-chsk krb5kdc[13053](info): commencing operation
krb5kdc: starting...
W20260810 14:39:59.769737 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:39:59 dist-test-slave-chsk krb5kdc[13053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:39:59.902896 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:00.019309 13066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:00.019639 13066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:00.019733 13066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:00.023797 13066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:00.023890 13066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:00.023938 13066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:00.023977 13066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:00.024009 13066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:00.029287 13066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:00.030386 13066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:00.031484 13066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:00.039960 13073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:00.040026 13075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:00.039976 13072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:00.040030 13066 server_base.cc:1034] running on GCE node
I20260810 14:40:00.040843 13066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:00.041935 13066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:00.043118 13066 hybrid_clock.cc:648] HybridClock initialized: now 1786372800043099 us; error 31 us; skew 500 ppm
Aug 10 14:40:00 dist-test-slave-chsk krb5kdc[13053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:00.046008 13066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:40:00.046959 13066 webserver.cc:460] Webserver started at http://127.0.0.1:33721/ using document root <none> and password file <none>
I20260810 14:40:00.047581 13066 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:00.047654 13066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:00.047873 13066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:00.049962 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "17a19d9401ba4dd6af6efdec71a7cfd0"
format_stamp: "Formatted at 2026-08-10 14:40:00 on dist-test-slave-chsk"
server_key: "595493db8198969f2298556d3e528d26"
server_key_iv: "a7526478df8ca91c5342fb92b2f74f5d"
server_key_version: "encryptionkey@0"
I20260810 14:40:00.050597 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "17a19d9401ba4dd6af6efdec71a7cfd0"
format_stamp: "Formatted at 2026-08-10 14:40:00 on dist-test-slave-chsk"
server_key: "595493db8198969f2298556d3e528d26"
server_key_iv: "a7526478df8ca91c5342fb92b2f74f5d"
server_key_version: "encryptionkey@0"
I20260810 14:40:00.054345 13066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 14:40:00.056741 13082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:00.057932 13066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:40:00.058076 13066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "17a19d9401ba4dd6af6efdec71a7cfd0"
format_stamp: "Formatted at 2026-08-10 14:40:00 on dist-test-slave-chsk"
server_key: "595493db8198969f2298556d3e528d26"
server_key_iv: "a7526478df8ca91c5342fb92b2f74f5d"
server_key_version: "encryptionkey@0"
I20260810 14:40:00.058187 13066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:00.062794 13066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:00.065582 13066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:00.065785 13066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:00.073904 13066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33683
I20260810 14:40:00.073931 13145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33683 every 8 connection(s)
I20260810 14:40:00.074819 13066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:00.077513 13146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:00.080673 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13066
I20260810 14:40:00.080818 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:00.081091 21677 external_mini_cluster.cc:1428] Setting key 737eb9f1abb2bcb508b27f471478a70c
I20260810 14:40:00.083482 13146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0: Bootstrap starting.
I20260810 14:40:00.086193 13146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:00.086756 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:00.082455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:46187 (local address 127.0.0.1:33683)
0810 14:40:00.082892 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:00.082910 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:00.082952 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:40:00.083792 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:00.083814 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:00.084142 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:00.084520 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:00.085200 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:00.085215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:00.085229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:00.085240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:00.085302 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:00.085326 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:00.085461 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:00.085557 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 14:40:00.085849 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260810 14:40:00.087075 13146 log.cc:824] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:00.089149 13146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0: No bootstrap required, opened a new log
I20260810 14:40:00.089582 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:00.093241 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:00.091640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:39429 (local address 127.0.0.1:33683)
0810 14:40:00.091763 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:00.091766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:00.091905 (+ 139us) server_negotiation.cc:407] Connection header received
0810 14:40:00.092032 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:00.092033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:00.092061 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:00.092126 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:00.092834 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:00.092836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:00.092839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:00.092849 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:00.092872 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:00.092882 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:00.092950 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:00.093069 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:40:00.093104 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:00.093381 13146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } }
I20260810 14:40:00.093740 13146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:00.093806 13146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a19d9401ba4dd6af6efdec71a7cfd0, State: Initialized, Role: FOLLOWER
I20260810 14:40:00.094275 13146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } }
I20260810 14:40:00.094396 13146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:00.094481 13146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:00.094578 13146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:00.095579 13146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } }
I20260810 14:40:00.095918 13146 leader_election.cc:302] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a19d9401ba4dd6af6efdec71a7cfd0; no voters:
I20260810 14:40:00.096226 13146 leader_election.cc:288] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:00.096323 13157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:00.096602 13157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [term 1 LEADER]: Becoming Leader. State: Replica: 17a19d9401ba4dd6af6efdec71a7cfd0, State: Running, Role: LEADER
I20260810 14:40:00.096956 13157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } }
I20260810 14:40:00.097452 13146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:00.099164 13159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a19d9401ba4dd6af6efdec71a7cfd0. Latest consensus state: current_term: 1 leader_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } } }
I20260810 14:40:00.099295 13159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:00.099303 13158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a19d9401ba4dd6af6efdec71a7cfd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33683 } } }
I20260810 14:40:00.099678 13158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:00.099692 13166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:00.102664 13166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:00.108235 13166 catalog_manager.cc:1347] Generated new cluster ID: 345cb889724046db884dbb92374f20c6
I20260810 14:40:00.108304 13166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:00.116227 13166 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:00.116530 13166 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:00.122442 13166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a19d9401ba4dd6af6efdec71a7cfd0: Generated new TSK 0
I20260810 14:40:00.123171 13166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:00.130749 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13066
2026-08-10T14:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:00 dist-test-slave-chsk krb5kdc[13182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:00 dist-test-slave-chsk krb5kdc[13182](info): set up 2 sockets
Aug 10 14:40:00 dist-test-slave-chsk krb5kdc[13182](info): commencing operation
krb5kdc: starting...
W20260810 14:40:01.227238 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:01.362528 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39071
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:33165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:01.476778 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:01.477061 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:01.477103 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:01.481117 13196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:01.481181 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:01.481201 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:01.481218 13196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:01.481235 13196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:01.486543 13196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:01.487725 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:01.488755 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:01.497624 13204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:01.497735 13202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:01.497772 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:01.497835 13196 server_base.cc:1034] running on GCE node
I20260810 14:40:01.498793 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:01.500002 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:01.501171 13196 hybrid_clock.cc:648] HybridClock initialized: now 1786372801501157 us; error 38 us; skew 500 ppm
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372801, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:01.503985 13196 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:01.504909 13196 webserver.cc:460] Webserver started at http://127.21.43.126:45291/ using document root <none> and password file <none>
I20260810 14:40:01.505450 13196 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:01.505510 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:01.505679 13196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:01.507800 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "bee571b195954d289cf7a00e3ee6f066"
format_stamp: "Formatted at 2026-08-10 14:40:01 on dist-test-slave-chsk"
server_key: "ade472091702ae0734c88fdeb173d796"
server_key_iv: "508f4c509f4d762cb95f431c8903e7d2"
server_key_version: "encryptionkey@0"
I20260810 14:40:01.508330 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "bee571b195954d289cf7a00e3ee6f066"
format_stamp: "Formatted at 2026-08-10 14:40:01 on dist-test-slave-chsk"
server_key: "ade472091702ae0734c88fdeb173d796"
server_key_iv: "508f4c509f4d762cb95f431c8903e7d2"
server_key_version: "encryptionkey@0"
I20260810 14:40:01.512048 13196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:40:01.514406 13211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:01.515710 13196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:01.515846 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "bee571b195954d289cf7a00e3ee6f066"
format_stamp: "Formatted at 2026-08-10 14:40:01 on dist-test-slave-chsk"
server_key: "ade472091702ae0734c88fdeb173d796"
server_key_iv: "508f4c509f4d762cb95f431c8903e7d2"
server_key_version: "encryptionkey@0"
I20260810 14:40:01.515954 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:01.528489 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:01.531558 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:01.531772 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:01.538982 13196 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:39071
I20260810 14:40:01.539008 13270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:39071 every 8 connection(s)
I20260810 14:40:01.540016 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:01.542865 13271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:01.548394 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066: Bootstrap starting.
I20260810 14:40:01.550191 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13196
I20260810 14:40:01.550365 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:01.550637 21677 external_mini_cluster.cc:1428] Setting key 87ce58233d28842d1ee2a5f49b59fdbc
I20260810 14:40:01.551072 13271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:01.551879 13271 log.cc:824] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:01.554175 13271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066: No bootstrap required, opened a new log
I20260810 14:40:01.558620 13271 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } }
I20260810 14:40:01.558954 13271 raft_consensus.cc:374] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:01.559016 13271 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee571b195954d289cf7a00e3ee6f066, State: Initialized, Role: FOLLOWER
I20260810 14:40:01.559530 13271 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } }
I20260810 14:40:01.559659 13271 raft_consensus.cc:388] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:01.559746 13271 raft_consensus.cc:482] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:01.559872 13271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:01.561187 13271 raft_consensus.cc:504] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } }
I20260810 14:40:01.561614 13271 leader_election.cc:302] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bee571b195954d289cf7a00e3ee6f066; no voters:
I20260810 14:40:01.562081 13271 leader_election.cc:288] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:01.562191 13276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:01.562461 13276 raft_consensus.cc:686] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [term 1 LEADER]: Becoming Leader. State: Replica: bee571b195954d289cf7a00e3ee6f066, State: Running, Role: LEADER
I20260810 14:40:01.562863 13276 consensus_queue.cc:237] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } }
I20260810 14:40:01.563581 13271 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:01.564637 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bee571b195954d289cf7a00e3ee6f066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } } }
I20260810 14:40:01.564778 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:01.565014 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bee571b195954d289cf7a00e3ee6f066. Latest consensus state: current_term: 1 leader_uuid: "bee571b195954d289cf7a00e3ee6f066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee571b195954d289cf7a00e3ee6f066" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39071 } } }
I20260810 14:40:01.565088 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:01.566177 13285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:01.566901 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:01.551787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50905 (local address 127.21.43.126:39071)
0810 14:40:01.552259 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:01.552276 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:01.552313 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:40:01.552957 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:01.552973 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:01.553322 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:01.553597 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:01.554556 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:01.555606 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:01.556832 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:01.557354 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:01.557476 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:01.560521 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:01.560528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:01.560545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:01.560577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:01.562640 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:01.563247 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:01.563253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:01.563259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:01.563402 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:01.563844 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:01.563848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:01.563850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:01.564381 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:01.565449 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:01.565653 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 14:40:01.565905 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260810 14:40:01.569657 13285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:01.570339 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:01.575736 13285 catalog_manager.cc:1347] Generated new cluster ID: e0991e44e2d64b1f909dbffefe66f941
I20260810 14:40:01.575800 13285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:01.582440 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:01.572574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38011 (local address 127.21.43.126:39071)
0810 14:40:01.572696 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:01.572710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:01.573055 (+ 345us) server_negotiation.cc:407] Connection header received
0810 14:40:01.573178 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:01.573181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:01.573231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:01.573323 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:01.574118 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:01.575433 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:01.576653 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:01.576995 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:01.577091 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:01.578820 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:01.578823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:01.578826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:01.578851 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:01.580570 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:01.581122 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:01.581125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:01.581127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:01.581205 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:01.581558 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:01.581560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:01.581562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:01.581755 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:01.581884 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:01.582192 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 14:40:01.582284 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:40:01.603278 13285 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:01.604671 13285 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:01.619397 13285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bee571b195954d289cf7a00e3ee6f066: Generated new TSK 0
I20260810 14:40:01.620381 13285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:01.628618 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13196
W20260810 14:40:01.637163 13178 connection.cc:535] client connection to 127.21.43.126:39071 recv error: Network error: failed to read from TLS socket (remote: 127.21.43.126:39071): Connection reset by peer (error 104)
2026-08-10T14:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13303](info): set up 2 sockets
Aug 10 14:40:01 dist-test-slave-chsk krb5kdc[13303](info): commencing operation
krb5kdc: starting...
W20260810 14:40:02.730825 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:02 dist-test-slave-chsk krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:02.869881 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34597
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:02.987548 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:02.987900 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:02.987988 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:02.992161 13318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:02.992277 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:02.992316 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:02.992389 13318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:02.992430 13318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:02.998095 13318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:34597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:02.999401 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:03.000523 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:03.009831 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:03.009831 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:03.009824 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:03.010253 13318 server_base.cc:1034] running on GCE node
I20260810 14:40:03.010972 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:03.012233 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:03.013429 13318 hybrid_clock.cc:648] HybridClock initialized: now 1786372803013421 us; error 34 us; skew 500 ppm
Aug 10 14:40:03 dist-test-slave-chsk krb5kdc[13303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372803, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:03.016502 13318 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:03.017601 13318 webserver.cc:460] Webserver started at http://127.21.43.126:40617/ using document root <none> and password file <none>
I20260810 14:40:03.018200 13318 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:03.018294 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:03.018544 13318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:03.020762 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "22ce30cd5ea34476985d857e25422299"
format_stamp: "Formatted at 2026-08-10 14:40:03 on dist-test-slave-chsk"
server_key: "7797b38f6e24fe5125e84eb300eb46cd"
server_key_iv: "67333ae10d0560a2dd47b96b62b10fc8"
server_key_version: "encryptionkey@0"
I20260810 14:40:03.021385 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "22ce30cd5ea34476985d857e25422299"
format_stamp: "Formatted at 2026-08-10 14:40:03 on dist-test-slave-chsk"
server_key: "7797b38f6e24fe5125e84eb300eb46cd"
server_key_iv: "67333ae10d0560a2dd47b96b62b10fc8"
server_key_version: "encryptionkey@0"
I20260810 14:40:03.025413 13318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:03.028025 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:03.029285 13318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:40:03.029439 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "22ce30cd5ea34476985d857e25422299"
format_stamp: "Formatted at 2026-08-10 14:40:03 on dist-test-slave-chsk"
server_key: "7797b38f6e24fe5125e84eb300eb46cd"
server_key_iv: "67333ae10d0560a2dd47b96b62b10fc8"
server_key_version: "encryptionkey@0"
I20260810 14:40:03.029564 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:03.043401 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:03.046732 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:03.046959 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:03.054895 13318 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:34597
I20260810 14:40:03.054970 13398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:34597 every 8 connection(s)
I20260810 14:40:03.056079 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:03.057926 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13318
I20260810 14:40:03.058111 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:03.058398 21677 external_mini_cluster.cc:1428] Setting key 5dbd99a5440ed47b0fc264992ac16ce7
I20260810 14:40:03.059427 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:03.066609 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299: Bootstrap starting.
I20260810 14:40:03.068581 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:03.059996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38105 (local address 127.21.43.126:34597)
0810 14:40:03.060456 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:03.060474 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:03.060514 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:40:03.061171 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:03.061190 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:03.061525 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:03.061811 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:03.062238 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:03.063355 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:03.064682 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:03.065190 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:03.065328 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:03.066574 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:03.066579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:03.066592 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:03.067089 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:03.067239 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:03.067271 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:03.067388 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:03.067593 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 14:40:03.067767 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":158,"threads_started":1}
I20260810 14:40:03.069427 13399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:03.070235 13399 log.cc:824] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:03.071942 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:03.072660 13399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299: No bootstrap required, opened a new log
I20260810 14:40:03.077589 13399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } }
I20260810 14:40:03.077970 13399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:03.078073 13399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ce30cd5ea34476985d857e25422299, State: Initialized, Role: FOLLOWER
I20260810 14:40:03.078631 13399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } }
I20260810 14:40:03.078784 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:03.074153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35717 (local address 127.21.43.126:34597)
0810 14:40:03.074283 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:03.074287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:03.074394 (+ 107us) server_negotiation.cc:407] Connection header received
0810 14:40:03.074521 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:03.074523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:03.074562 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:03.074629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:03.074959 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:03.075542 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:03.076715 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:03.076959 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:03.077075 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:03.078102 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:03.078106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:03.078108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:03.078138 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:03.078173 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:03.078192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:03.078268 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:03.078479 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:40:03.078557 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:40:03.078783 13399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:03.078954 13399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:03.079073 13399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:03.080484 13399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } }
I20260810 14:40:03.080899 13399 leader_election.cc:302] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ce30cd5ea34476985d857e25422299; no voters:
I20260810 14:40:03.081324 13399 leader_election.cc:288] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:03.081491 13411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:03.081794 13411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [term 1 LEADER]: Becoming Leader. State: Replica: 22ce30cd5ea34476985d857e25422299, State: Running, Role: LEADER
I20260810 14:40:03.082281 13411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } }
I20260810 14:40:03.082881 13399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:03.084131 13413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ce30cd5ea34476985d857e25422299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } } }
I20260810 14:40:03.084270 13413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:03.084649 13420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:03.084692 13412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ce30cd5ea34476985d857e25422299. Latest consensus state: current_term: 1 leader_uuid: "22ce30cd5ea34476985d857e25422299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ce30cd5ea34476985d857e25422299" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 34597 } } }
I20260810 14:40:03.085407 13412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:03.088120 13420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:03.094413 13420 catalog_manager.cc:1347] Generated new cluster ID: e8016da22d0f459aba4dddea2c28f2b6
I20260810 14:40:03.094565 13420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:03.103257 13420 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:03.104235 13420 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:03.113201 13420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ce30cd5ea34476985d857e25422299: Generated new TSK 0
I20260810 14:40:03.113960 13420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:03.125855 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13318
W20260810 14:40:03.135120 13301 connection.cc:535] client connection to 127.21.43.126:34597 recv error: Network error: failed to read from TLS socket (remote: 127.21.43.126:34597): Connection reset by peer (error 104)
2026-08-10T14:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:03 dist-test-slave-chsk krb5kdc[13433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:03 dist-test-slave-chsk krb5kdc[13433](info): set up 2 sockets
Aug 10 14:40:03 dist-test-slave-chsk krb5kdc[13433](info): commencing operation
krb5kdc: starting...
W20260810 14:40:04.235812 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:04 dist-test-slave-chsk krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:04.370095 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40573
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:40573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:04.489420 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:04.489780 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:04.489867 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:04.494127 13449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:04.494241 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:04.494275 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:04.494318 13449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:04.494352 13449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:04.500823 13449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:40573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:40573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:04.502147 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:04.503302 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:04.512728 13455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:04.512744 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:04.512728 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:04.513123 13449 server_base.cc:1034] running on GCE node
I20260810 14:40:04.513762 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:04.514958 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:04.516145 13449 hybrid_clock.cc:648] HybridClock initialized: now 1786372804516135 us; error 35 us; skew 500 ppm
Aug 10 14:40:04 dist-test-slave-chsk krb5kdc[13433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372804, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:04.524298 13449 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:04.525367 13449 webserver.cc:460] Webserver started at http://127.21.43.126:37619/ using document root <none> and password file <none>
I20260810 14:40:04.525933 13449 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:04.525998 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:04.526185 13449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:04.528475 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "d2242a82763e41d39966094eb41d7a87"
format_stamp: "Formatted at 2026-08-10 14:40:04 on dist-test-slave-chsk"
server_key: "c7f5afbb2b0b35f9ccee3346658704fb"
server_key_iv: "9f5a93b524e9a78db4bb860a4e35d812"
server_key_version: "encryptionkey@0"
I20260810 14:40:04.529022 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "d2242a82763e41d39966094eb41d7a87"
format_stamp: "Formatted at 2026-08-10 14:40:04 on dist-test-slave-chsk"
server_key: "c7f5afbb2b0b35f9ccee3346658704fb"
server_key_iv: "9f5a93b524e9a78db4bb860a4e35d812"
server_key_version: "encryptionkey@0"
I20260810 14:40:04.533000 13449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:04.535805 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:04.537096 13449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:04.537256 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "d2242a82763e41d39966094eb41d7a87"
format_stamp: "Formatted at 2026-08-10 14:40:04 on dist-test-slave-chsk"
server_key: "c7f5afbb2b0b35f9ccee3346658704fb"
server_key_iv: "9f5a93b524e9a78db4bb860a4e35d812"
server_key_version: "encryptionkey@0"
I20260810 14:40:04.537384 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:04.542229 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:04.545378 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:04.545540 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:04.553725 13449 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:40573
I20260810 14:40:04.553752 13522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:40573 every 8 connection(s)
I20260810 14:40:04.554780 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:04.557678 13523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:04.558082 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13449
I20260810 14:40:04.558245 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:04.558549 21677 external_mini_cluster.cc:1428] Setting key eddf859101211fd3e6c4196c4fad2ed1
I20260810 14:40:04.563877 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:04.559931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46125 (local address 127.21.43.126:40573)
0810 14:40:04.560385 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:04.560401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:04.560438 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:40:04.561113 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:04.561132 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:04.561429 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:04.561739 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:04.562610 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:04.562623 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:04.562635 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:04.562645 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:04.562698 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:04.562718 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:04.562831 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:04.562951 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:40:04.563124 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
I20260810 14:40:04.565027 13523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87: Bootstrap starting.
I20260810 14:40:04.567072 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:04.567821 13523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:04.568661 13523 log.cc:824] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:04.570776 13523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87: No bootstrap required, opened a new log
I20260810 14:40:04.571312 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:04.569450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36493 (local address 127.21.43.126:40573)
0810 14:40:04.569588 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:04.569592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:04.569818 (+ 226us) server_negotiation.cc:407] Connection header received
0810 14:40:04.569982 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:04.569985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:04.570026 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:04.570113 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:04.570864 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:04.570868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:04.570874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:04.570887 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:04.570918 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:04.570931 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:04.571032 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:04.571118 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 14:40:04.571184 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:40:04.574955 13523 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } }
I20260810 14:40:04.575268 13523 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:04.575346 13523 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2242a82763e41d39966094eb41d7a87, State: Initialized, Role: FOLLOWER
I20260810 14:40:04.575810 13523 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } }
I20260810 14:40:04.575958 13523 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:04.576030 13523 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:04.576170 13523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:04.577181 13523 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } }
I20260810 14:40:04.577546 13523 leader_election.cc:302] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2242a82763e41d39966094eb41d7a87; no voters:
I20260810 14:40:04.577889 13523 leader_election.cc:288] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:04.578032 13535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:04.578341 13535 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [term 1 LEADER]: Becoming Leader. State: Replica: d2242a82763e41d39966094eb41d7a87, State: Running, Role: LEADER
I20260810 14:40:04.578691 13535 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } }
I20260810 14:40:04.579308 13523 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:04.580924 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2242a82763e41d39966094eb41d7a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } } }
I20260810 14:40:04.580978 13537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2242a82763e41d39966094eb41d7a87. Latest consensus state: current_term: 1 leader_uuid: "d2242a82763e41d39966094eb41d7a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2242a82763e41d39966094eb41d7a87" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 40573 } } }
I20260810 14:40:04.581072 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:04.581097 13537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:04.582064 13547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:04.585386 13547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:04.591502 13547 catalog_manager.cc:1347] Generated new cluster ID: 3b4c7e44ffdb4afaa1c66b9323e9b92e
I20260810 14:40:04.591598 13547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:04.605585 13547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:04.605947 13547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:04.613000 13547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2242a82763e41d39966094eb41d7a87: Generated new TSK 0
I20260810 14:40:04.613961 13547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:04.627620 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13449
2026-08-10T14:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:04 dist-test-slave-chsk krb5kdc[13556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:04 dist-test-slave-chsk krb5kdc[13556](info): set up 2 sockets
Aug 10 14:40:04 dist-test-slave-chsk krb5kdc[13556](info): commencing operation
krb5kdc: starting...
W20260810 14:40:05.756886 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:05 dist-test-slave-chsk krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:05.888024 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:36671
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:36671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:06.001965 13571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:06.002272 13571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:06.002348 13571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:06.006482 13571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:06.006551 13571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:06.006572 13571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:06.006618 13571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:06.006659 13571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:06.012228 13571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:36671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:36671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:06.013331 13571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:06.014374 13571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:06.023034 13579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:06.023034 13577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:06.023046 13576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:06.023512 13571 server_base.cc:1034] running on GCE node
I20260810 14:40:06.024075 13571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:06.025208 13571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:06.026408 13571 hybrid_clock.cc:648] HybridClock initialized: now 1786372806026394 us; error 36 us; skew 500 ppm
Aug 10 14:40:06 dist-test-slave-chsk krb5kdc[13556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372806, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:06.029253 13571 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:06.030156 13571 webserver.cc:460] Webserver started at http://127.21.43.126:44181/ using document root <none> and password file <none>
I20260810 14:40:06.030725 13571 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:06.030819 13571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:06.031059 13571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:06.033191 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "d16b0a40cd84407caf6830f62b983e13"
format_stamp: "Formatted at 2026-08-10 14:40:06 on dist-test-slave-chsk"
server_key: "78e831e042163976af7ae17aa26fe6f0"
server_key_iv: "4c2bad13265bd5e4aa5c8485acce1ae0"
server_key_version: "encryptionkey@0"
I20260810 14:40:06.033769 13571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "d16b0a40cd84407caf6830f62b983e13"
format_stamp: "Formatted at 2026-08-10 14:40:06 on dist-test-slave-chsk"
server_key: "78e831e042163976af7ae17aa26fe6f0"
server_key_iv: "4c2bad13265bd5e4aa5c8485acce1ae0"
server_key_version: "encryptionkey@0"
I20260810 14:40:06.037636 13571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 14:40:06.040066 13587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:06.041270 13571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:06.041419 13571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "d16b0a40cd84407caf6830f62b983e13"
format_stamp: "Formatted at 2026-08-10 14:40:06 on dist-test-slave-chsk"
server_key: "78e831e042163976af7ae17aa26fe6f0"
server_key_iv: "4c2bad13265bd5e4aa5c8485acce1ae0"
server_key_version: "encryptionkey@0"
I20260810 14:40:06.041533 13571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:06.060500 13571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:06.063690 13571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:06.063910 13571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:06.071333 13571 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:36671
I20260810 14:40:06.071421 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:36671 every 8 connection(s)
I20260810 14:40:06.072358 13571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:06.075181 13644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:06.075397 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13571
I20260810 14:40:06.075557 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:06.075858 21677 external_mini_cluster.cc:1428] Setting key 52c21bca683c135c8550cb508845ccda
I20260810 14:40:06.082530 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13: Bootstrap starting.
I20260810 14:40:06.085314 13644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13: Neither blocks nor log segments found. Creating new log.
Aug 10 14:40:06 dist-test-slave-chsk krb5kdc[13556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:06.086254 13644 log.cc:824] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:06.088642 13644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13: No bootstrap required, opened a new log
I20260810 14:40:06.091014 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:06.077227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45265 (local address 127.21.43.126:36671)
0810 14:40:06.077669 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:06.077689 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:06.077732 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:40:06.078625 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:06.078650 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:06.079083 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:06.079474 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:06.080178 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:06.081687 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:06.082530 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:06.083207 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:06.083358 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:06.086160 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:06.086165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:06.086178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:06.086207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:06.088111 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:06.088631 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:06.088635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:06.088639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:06.088815 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:06.089130 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:06.089132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:06.089133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:06.089514 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:06.089659 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:06.089976 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 14:40:06.090282 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260810 14:40:06.093124 13644 raft_consensus.cc:348] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } }
I20260810 14:40:06.093479 13644 raft_consensus.cc:374] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:06.093547 13644 raft_consensus.cc:729] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16b0a40cd84407caf6830f62b983e13, State: Initialized, Role: FOLLOWER
I20260810 14:40:06.094030 13644 consensus_queue.cc:260] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } }
I20260810 14:40:06.094193 13644 raft_consensus.cc:388] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:06.094264 13644 raft_consensus.cc:482] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:06.094367 13644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:06.095198 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:06.095614 13644 raft_consensus.cc:504] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } }
I20260810 14:40:06.095965 13644 leader_election.cc:302] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d16b0a40cd84407caf6830f62b983e13; no voters:
I20260810 14:40:06.096369 13644 leader_election.cc:288] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:06.096485 13649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:06.096706 13649 raft_consensus.cc:686] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [term 1 LEADER]: Becoming Leader. State: Replica: d16b0a40cd84407caf6830f62b983e13, State: Running, Role: LEADER
I20260810 14:40:06.097012 13649 consensus_queue.cc:237] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } }
I20260810 14:40:06.097785 13644 sys_catalog.cc:564] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:06.098197 13649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d16b0a40cd84407caf6830f62b983e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } } }
I20260810 14:40:06.098369 13649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:06.098635 13649 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d16b0a40cd84407caf6830f62b983e13. Latest consensus state: current_term: 1 leader_uuid: "d16b0a40cd84407caf6830f62b983e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d16b0a40cd84407caf6830f62b983e13" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 36671 } } }
I20260810 14:40:06.098824 13649 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:06.099192 13663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:06.102633 13663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:06.106959 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:06.097515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55319 (local address 127.21.43.126:36671)
0810 14:40:06.097629 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:06.097640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:06.097970 (+ 330us) server_negotiation.cc:407] Connection header received
0810 14:40:06.098135 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:06.098138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:06.098189 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:06.098242 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:06.099272 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:06.100048 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:06.101300 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:06.101553 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:06.101680 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:06.103516 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:06.103519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:06.103522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:06.103548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:06.105081 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:06.105591 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:06.105594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:06.105597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:06.105691 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:06.106011 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:06.106014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:06.106016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:06.106248 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:06.106394 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:06.106657 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 14:40:06.106789 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:40:06.109615 13663 catalog_manager.cc:1347] Generated new cluster ID: 4699f003ea864854885014061cc3d3c4
I20260810 14:40:06.109679 13663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:06.126559 13663 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:06.127640 13663 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:06.134526 13663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d16b0a40cd84407caf6830f62b983e13: Generated new TSK 0
I20260810 14:40:06.135349 13663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:06.145715 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13571
2026-08-10T14:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:06 dist-test-slave-chsk krb5kdc[13682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:06 dist-test-slave-chsk krb5kdc[13682](info): set up 2 sockets
Aug 10 14:40:06 dist-test-slave-chsk krb5kdc[13682](info): commencing operation
krb5kdc: starting...
W20260810 14:40:07.238557 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:07 dist-test-slave-chsk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:07.369863 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37541
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:37541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:07.482618 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:07.482931 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:07.483017 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:07.487051 13696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:07.487128 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:07.487181 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:07.487226 13696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:07.487295 13696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:07.492895 13696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:37541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:07.493860 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:07.494812 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:07.503273 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:07.503278 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:07.503345 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:07.503345 13696 server_base.cc:1034] running on GCE node
I20260810 14:40:07.503995 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:07.505084 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:07.506278 13696 hybrid_clock.cc:648] HybridClock initialized: now 1786372807506251 us; error 39 us; skew 500 ppm
Aug 10 14:40:07 dist-test-slave-chsk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372807, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:07.509061 13696 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:07.509963 13696 webserver.cc:460] Webserver started at http://127.21.43.126:38829/ using document root <none> and password file <none>
I20260810 14:40:07.510519 13696 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:07.510612 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:07.510844 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:07.512965 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "ee85f7bad521485ebd06d4bc81a2acb9"
format_stamp: "Formatted at 2026-08-10 14:40:07 on dist-test-slave-chsk"
server_key: "26abc192aa611aa521d8b82bb7c9326a"
server_key_iv: "fce440c75be11e3926be6a9ae7232ff2"
server_key_version: "encryptionkey@0"
I20260810 14:40:07.513536 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "ee85f7bad521485ebd06d4bc81a2acb9"
format_stamp: "Formatted at 2026-08-10 14:40:07 on dist-test-slave-chsk"
server_key: "26abc192aa611aa521d8b82bb7c9326a"
server_key_iv: "fce440c75be11e3926be6a9ae7232ff2"
server_key_version: "encryptionkey@0"
I20260810 14:40:07.517445 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:07.520015 13713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:07.521238 13696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:07.521389 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "ee85f7bad521485ebd06d4bc81a2acb9"
format_stamp: "Formatted at 2026-08-10 14:40:07 on dist-test-slave-chsk"
server_key: "26abc192aa611aa521d8b82bb7c9326a"
server_key_iv: "fce440c75be11e3926be6a9ae7232ff2"
server_key_version: "encryptionkey@0"
I20260810 14:40:07.521508 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:07.552268 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:07.555902 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:07.556135 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:07.563777 13696 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:37541
I20260810 14:40:07.563805 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:37541 every 8 connection(s)
I20260810 14:40:07.564746 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:07.567317 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13696
I20260810 14:40:07.567464 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:07.567651 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:07.567770 21677 external_mini_cluster.cc:1428] Setting key 0c81ebb8804b308f0bf292019de31840
I20260810 14:40:07.575234 13770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9: Bootstrap starting.
I20260810 14:40:07.578079 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:07.569238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42989 (local address 127.21.43.126:37541)
0810 14:40:07.569690 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:07.569709 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:07.569752 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:40:07.570627 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:07.570651 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:07.571093 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:07.571490 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:07.571906 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:07.573381 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:07.574279 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:07.574930 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:07.575094 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:07.575975 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:07.575981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:07.575997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:07.576513 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:07.576590 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:07.576622 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:07.576734 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:07.576853 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:40:07.577073 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":124,"threads_started":1}
I20260810 14:40:07.578225 13770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:07.578998 13770 log.cc:824] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:07.581179 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:07.581221 13770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9: No bootstrap required, opened a new log
I20260810 14:40:07.585261 13770 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } }
I20260810 14:40:07.585532 13770 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:07.585631 13770 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee85f7bad521485ebd06d4bc81a2acb9, State: Initialized, Role: FOLLOWER
I20260810 14:40:07.586099 13770 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } }
I20260810 14:40:07.586274 13770 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:07.586347 13770 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:07.586464 13770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:07.587213 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:07.583299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36123 (local address 127.21.43.126:37541)
0810 14:40:07.583438 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:07.583442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:07.583567 (+ 125us) server_negotiation.cc:407] Connection header received
0810 14:40:07.583734 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:07.583737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:07.583789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:07.583872 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:07.584131 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:07.584648 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:07.585440 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:07.585784 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:07.585895 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:07.586764 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:07.586768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:07.586771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:07.586783 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:07.586818 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:07.586829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:07.586910 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:07.586999 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 14:40:07.587053 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:40:07.587656 13770 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } }
I20260810 14:40:07.588032 13770 leader_election.cc:302] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee85f7bad521485ebd06d4bc81a2acb9; no voters:
I20260810 14:40:07.588408 13770 leader_election.cc:288] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:07.588500 13781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:07.588757 13781 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [term 1 LEADER]: Becoming Leader. State: Replica: ee85f7bad521485ebd06d4bc81a2acb9, State: Running, Role: LEADER
I20260810 14:40:07.589085 13781 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } }
I20260810 14:40:07.589727 13770 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:07.590755 13782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } } }
I20260810 14:40:07.590883 13782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:07.591216 13784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee85f7bad521485ebd06d4bc81a2acb9. Latest consensus state: current_term: 1 leader_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85f7bad521485ebd06d4bc81a2acb9" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 37541 } } }
I20260810 14:40:07.591310 13784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:07.591418 13792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:07.594607 13792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:07.604384 13792 catalog_manager.cc:1347] Generated new cluster ID: 35d973ff9e804599a314bd7a26739cd7
I20260810 14:40:07.604493 13792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:07.624656 13792 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:07.625936 13792 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:07.632329 13792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee85f7bad521485ebd06d4bc81a2acb9: Generated new TSK 0
I20260810 14:40:07.633167 13792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:07.638592 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13696
2026-08-10T14:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:07 dist-test-slave-chsk krb5kdc[13804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:07 dist-test-slave-chsk krb5kdc[13804](info): set up 2 sockets
Aug 10 14:40:07 dist-test-slave-chsk krb5kdc[13804](info): commencing operation
krb5kdc: starting...
W20260810 14:40:08.739388 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:08 dist-test-slave-chsk krb5kdc[13804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:08.874579 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41723
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:08.990347 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:08.990712 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:08.990803 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:08.994962 13818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:08.995085 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:08.995124 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:08.995201 13818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:08.995242 13818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:09.000826 13818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:09.002105 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:09.003228 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:09.011924 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:09.011926 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:09.011947 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:09.012377 13818 server_base.cc:1034] running on GCE node
I20260810 14:40:09.012917 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:09.014046 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:09.015255 13818 hybrid_clock.cc:648] HybridClock initialized: now 1786372809015230 us; error 34 us; skew 500 ppm
Aug 10 14:40:09 dist-test-slave-chsk krb5kdc[13804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372809, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:09.018250 13818 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:09.019227 13818 webserver.cc:460] Webserver started at http://127.21.43.126:42969/ using document root <none> and password file <none>
I20260810 14:40:09.019814 13818 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:09.019908 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:09.020159 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:09.022470 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "1fd95805ee824d4698afb42256bcf213"
format_stamp: "Formatted at 2026-08-10 14:40:09 on dist-test-slave-chsk"
server_key: "59876aaec4f7abd7a7d290faedb43829"
server_key_iv: "e6755ed69ff98b7d109bcdc4b83d59b9"
server_key_version: "encryptionkey@0"
I20260810 14:40:09.023196 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "1fd95805ee824d4698afb42256bcf213"
format_stamp: "Formatted at 2026-08-10 14:40:09 on dist-test-slave-chsk"
server_key: "59876aaec4f7abd7a7d290faedb43829"
server_key_iv: "e6755ed69ff98b7d109bcdc4b83d59b9"
server_key_version: "encryptionkey@0"
I20260810 14:40:09.027064 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:40:09.029635 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:09.030872 13818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:40:09.031026 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "1fd95805ee824d4698afb42256bcf213"
format_stamp: "Formatted at 2026-08-10 14:40:09 on dist-test-slave-chsk"
server_key: "59876aaec4f7abd7a7d290faedb43829"
server_key_iv: "e6755ed69ff98b7d109bcdc4b83d59b9"
server_key_version: "encryptionkey@0"
I20260810 14:40:09.031184 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:09.036048 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:09.039274 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:09.039484 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:09.047382 13818 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:41723
I20260810 14:40:09.047430 13893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:41723 every 8 connection(s)
I20260810 14:40:09.048437 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:09.051723 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:09.052498 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13818
I20260810 14:40:09.052687 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:09.052980 21677 external_mini_cluster.cc:1428] Setting key 73ad4084eedd81fd8df8bad0c79e1203
I20260810 14:40:09.059705 13894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: Bootstrap starting.
I20260810 14:40:09.061017 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:09.055412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54451 (local address 127.21.43.126:41723)
0810 14:40:09.056106 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:09.056190 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:09.056271 (+ 81us) server_negotiation.cc:407] Connection header received
0810 14:40:09.057434 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:09.057462 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:09.057913 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:09.058414 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:09.059502 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:09.059508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:09.059523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:09.059535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:09.059607 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:09.059631 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:09.059825 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:09.060030 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 14:40:09.060236 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":157,"threads_started":1}
I20260810 14:40:09.062664 13894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:09.063616 13894 log.cc:824] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:09.064502 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:09.066082 13894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: No bootstrap required, opened a new log
I20260810 14:40:09.069142 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:09.067192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52493 (local address 127.21.43.126:41723)
0810 14:40:09.067326 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:09.067330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:09.067570 (+ 240us) server_negotiation.cc:407] Connection header received
0810 14:40:09.067736 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:09.067739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:09.067784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:09.067875 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:09.068674 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:09.068678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:09.068681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:40:09.068693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:09.068731 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:40:09.068743 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:09.068842 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:09.068955 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 14:40:09.068997 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:40:09.070876 13894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } }
I20260810 14:40:09.071134 13894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:40:09.071265 13894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd95805ee824d4698afb42256bcf213, State: Initialized, Role: FOLLOWER
I20260810 14:40:09.071735 13894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } }
I20260810 14:40:09.071869 13894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:40:09.071915 13894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:40:09.072053 13894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:09.073190 13894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } }
I20260810 14:40:09.073612 13894 leader_election.cc:302] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fd95805ee824d4698afb42256bcf213; no voters:
I20260810 14:40:09.074002 13894 leader_election.cc:288] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:40:09.074425 13904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:09.074846 13904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [term 1 LEADER]: Becoming Leader. State: Replica: 1fd95805ee824d4698afb42256bcf213, State: Running, Role: LEADER
I20260810 14:40:09.075381 13904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } }
I20260810 14:40:09.075611 13894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:09.077857 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fd95805ee824d4698afb42256bcf213. Latest consensus state: current_term: 1 leader_uuid: "1fd95805ee824d4698afb42256bcf213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } } }
I20260810 14:40:09.078022 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:09.078296 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fd95805ee824d4698afb42256bcf213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fd95805ee824d4698afb42256bcf213" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 41723 } } }
I20260810 14:40:09.078543 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213 [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:09.081168 13922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:09.081270 13922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:09.081349 13923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:09.082193 13923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:09.089221 13923 catalog_manager.cc:1347] Generated new cluster ID: 34c0515608104287a0d3696000417cc5
I20260810 14:40:09.089329 13923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:09.130429 13923 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:09.130826 13923 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:09.145256 13923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fd95805ee824d4698afb42256bcf213: Generated new TSK 0
I20260810 14:40:09.146067 13923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:40:09.154168 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13818
2026-08-10T14:40:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1516 ms)
[----------] 18 tests from AuthTokenIssuingTest (26983 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:09 dist-test-slave-chsk krb5kdc[13929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:09 dist-test-slave-chsk krb5kdc[13929](info): set up 2 sockets
Aug 10 14:40:09 dist-test-slave-chsk krb5kdc[13929](info): commencing operation
krb5kdc: starting...
W20260810 14:40:10.262109 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:10.394079 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43811
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:10.509979 13946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:10.510375 13946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:10.510452 13946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:10.510536 13946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:10.514760 13946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:10.514873 13946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:10.514909 13946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:10.514950 13946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:10.515007 13946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:10.520816 13946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:10.522082 13946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:10.523236 13946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:10.532619 13953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:10.532620 13952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:10.532706 13946 server_base.cc:1034] running on GCE node
W20260810 14:40:10.532665 13956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:10.533517 13946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:10.534641 13946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:10.535844 13946 hybrid_clock.cc:648] HybridClock initialized: now 1786372810535805 us; error 61 us; skew 500 ppm
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:10.538805 13946 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:10.539856 13946 webserver.cc:460] Webserver started at http://127.21.43.126:40973/ using document root <none> and password file <none>
I20260810 14:40:10.540467 13946 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:10.540570 13946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:10.540807 13946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:10.542982 13946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "14010e5fec56475ab4334aee036e8f54"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "811dbf82af31f3a44fc26e1c09ac4cf3"
server_key_iv: "01aa33b89b18205431c8901449edfac8"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.543684 13946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "14010e5fec56475ab4334aee036e8f54"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "811dbf82af31f3a44fc26e1c09ac4cf3"
server_key_iv: "01aa33b89b18205431c8901449edfac8"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.547642 13946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:10.550220 13963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:10.551679 13946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:10.551836 13946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "14010e5fec56475ab4334aee036e8f54"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "811dbf82af31f3a44fc26e1c09ac4cf3"
server_key_iv: "01aa33b89b18205431c8901449edfac8"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.551964 13946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:10.568599 13946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:10.571914 13946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:10.572150 13946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:10.579671 13946 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:43811
I20260810 14:40:10.579988 14021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:43811 every 8 connection(s)
I20260810 14:40:10.580801 13946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:10.581578 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 13946
I20260810 14:40:10.581724 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:10.582082 21677 external_mini_cluster.cc:1428] Setting key ab3795a8851bd98e65e84436238666d9
I20260810 14:40:10.584456 14023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:10.588619 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.125@KRBTEST.COM" created.
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:10.605764 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.591233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43811 (local address 127.0.0.1:44936)
0810 14:40:10.591812 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.591851 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:10.592752 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:10.593585 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:10.593589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:10.594065 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:10.594422 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.594443 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.595680 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.595686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.596695 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.596706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.597358 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.597362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.597476 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.598110 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:10.598127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:10.599959 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:10.602446 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.602455 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.602464 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.602722 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.603127 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.603130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.603132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.603309 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.604119 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:10.604125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:10.604286 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 14:40:10.604486 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 14:40:10.604952 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":86,"thread_start_us":205,"threads_started":1}
I20260810 14:40:10.605769 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.591232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.21.43.126:43811)
0810 14:40:10.591733 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:10.591752 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:10.591964 (+ 212us) server_negotiation.cc:407] Connection header received
0810 14:40:10.593157 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:10.593166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:10.593378 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:10.593538 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:10.594598 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.595549 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.596823 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.597192 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.597365 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.600150 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:10.600160 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:10.600179 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:10.600215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:10.602316 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.602847 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.602855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.602858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.603009 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.603412 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.603416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.603417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.603981 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:10.604100 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:10.604587 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 14:40:10.604838 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260810 14:40:10.608362 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:10.609249 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.608937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57784)
0810 14:40:10.609099 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.609175 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 14:40:10.609464 13966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
I20260810 14:40:10.610514 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.610218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57796)
0810 14:40:10.610358 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.610409 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 14:40:10.610749 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37123: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
I20260810 14:40:10.612146 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:37123
--webserver_interface=127.21.43.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:10.660040 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } attempt: 1
I20260810 14:40:10.661023 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.660651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57802)
0810 14:40:10.660839 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.660917 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 14:40:10.661784 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.661550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57812)
0810 14:40:10.661659 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.661713 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:40:10.661935 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37123: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
I20260810 14:40:10.730163 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } attempt: 2
W20260810 14:40:10.730484 14030 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:10.730818 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:10.730870 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:10.730924 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:40:10.731103 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.730735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57816)
0810 14:40:10.730923 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.731010 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:40:10.731904 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.731667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57822)
0810 14:40:10.731783 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.731837 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:40:10.732071 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:37123: Network error: Client connection negotiation failed: client connection to 127.21.43.125:37123: connect: Connection refused (error 111)
W20260810 14:40:10.735620 14030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:10.735703 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:10.735749 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:10.735806 14030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:10.735844 14030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:10.741546 14030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.43.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:10.742731 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:10.743860 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:10.752562 14039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:10.752562 14037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:10.752586 14036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:10.752992 14030 server_base.cc:1034] running on GCE node
I20260810 14:40:10.753592 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:10.754753 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:10.755978 14030 hybrid_clock.cc:648] HybridClock initialized: now 1786372810755953 us; error 38 us; skew 500 ppm
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:10.759227 14030 init.cc:377] Logged in from keytab as kudu/127.21.43.125@KRBTEST.COM (short username kudu)
I20260810 14:40:10.760236 14030 webserver.cc:460] Webserver started at http://127.21.43.125:43789/ using document root <none> and password file <none>
I20260810 14:40:10.760819 14030 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:10.760912 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:10.761157 14030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:10.763403 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data/instance:
uuid: "18d7c97659fa45668dbd4025e2befdac"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "96dfc72cadf72367be7ea1d1d9859d33"
server_key_iv: "8987444eacf145be45fc229262cbda14"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.764006 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal/instance:
uuid: "18d7c97659fa45668dbd4025e2befdac"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "96dfc72cadf72367be7ea1d1d9859d33"
server_key_iv: "8987444eacf145be45fc229262cbda14"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.767928 14030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:40:10.770627 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:10.772029 14030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:40:10.772240 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "18d7c97659fa45668dbd4025e2befdac"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "96dfc72cadf72367be7ea1d1d9859d33"
server_key_iv: "8987444eacf145be45fc229262cbda14"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.772374 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:10.793155 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:10.796761 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:10.796995 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:10.804850 14030 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.125:37123
I20260810 14:40:10.805207 14109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.125:37123 every 8 connection(s)
I20260810 14:40:10.806003 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
I20260810 14:40:10.808969 14110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:10.810105 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 14030
I20260810 14:40:10.810250 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-1/wal/instance
I20260810 14:40:10.810565 21677 external_mini_cluster.cc:1428] Setting key bcf5ed0687dd094d94548bfbf3afb719
I20260810 14:40:10.813179 14110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.124@KRBTEST.COM" created.
I20260810 14:40:10.823215 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } attempt: 3
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:10.829607 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.815492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.21.43.126:43811)
0810 14:40:10.815688 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:10.815693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:10.816681 (+ 988us) server_negotiation.cc:407] Connection header received
0810 14:40:10.817619 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:10.817623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:10.817692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:10.817808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:10.818971 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.819889 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.821343 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.821610 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.821708 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.824867 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:10.824873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:10.824880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:10.824909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:10.827051 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.827738 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.827741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.827743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.827835 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.828197 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.828202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.828203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.828434 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:10.828836 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:10.829262 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 14:40:10.829407 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:40:10.830863 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.815955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43811 (local address 127.0.0.1:44938)
0810 14:40:10.816484 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.816537 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:10.817404 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:10.817962 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:10.817969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:10.818313 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:10.818799 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.818813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.820027 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.820029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.821204 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.821215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.821760 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.821763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.821910 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.822637 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:10.822659 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:10.824667 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:10.827205 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.827212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.827228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.827625 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.827931 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.827934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.827936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.828094 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.828574 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:10.828583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:10.828965 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 14:40:10.829211 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 14:40:10.829609 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260810 14:40:10.832599 14110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:10.835494 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.823708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57834)
0810 14:40:10.823899 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.823911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:10.823992 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:10.824944 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:10.824950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:10.825233 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:10.825378 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.825388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.826677 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.826682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.827381 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.827393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.828296 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.828299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.828409 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.828875 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:10.828889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:10.830921 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:10.833674 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.833678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.833680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.833885 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.834245 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.834251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.834254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.834364 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.834914 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:10.834917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:10.835026 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 14:40:10.835238 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 14:40:10.835372 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 14:40:10.835631 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.824202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.21.43.125:37123)
0810 14:40:10.824521 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:10.824537 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:10.824558 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:40:10.824594 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:10.824599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:10.824763 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:10.824936 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:10.825528 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.826539 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.827535 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.828175 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.828298 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.831467 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:10.831472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:10.831481 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:10.831503 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:10.833508 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.834028 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.834037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.834039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.834130 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.834493 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.834499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.834503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.834799 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:10.834906 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:10.835300 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 14:40:10.835448 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":129,"threads_started":1}
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:40:10.837981 14023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:10.838876 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.838579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46606)
0810 14:40:10.838726 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.838786 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:40:10.839694 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.839397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46612)
0810 14:40:10.839551 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.839596 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 14:40:10.839905 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
I20260810 14:40:10.842217 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.833173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57840)
0810 14:40:10.833302 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.833319 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:10.833405 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:10.833984 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:10.833990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:10.834183 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:10.834312 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.834317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.835137 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.835142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.835935 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:10.835948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.836393 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.836397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:10.836498 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.836956 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:10.836973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:10.838414 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:10.840629 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.840632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.840633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.840817 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.841192 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:10.841194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:10.841196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:10.841272 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:10.841706 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:10.841712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:10.841833 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 14:40:10.841955 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 14:40:10.842106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 14:40:10.842374 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.833376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.21.43.125:37123)
0810 14:40:10.833720 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:10.833733 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:10.833749 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:40:10.833788 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:10.833793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:10.833846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:10.833939 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:10.834417 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.835020 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.836042 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:10.836280 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:10.836364 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:10.838599 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:10.838602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:10.838626 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:10.838656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:10.840514 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.840994 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.840998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.841000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.841100 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:10.841384 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:10.841387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:10.841388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:10.841574 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:10.841722 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:10.842068 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 14:40:10.842198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260810 14:40:10.842730 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:32917
--webserver_interface=127.21.43.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:10.843070 14110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:10.843793 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.843545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46628)
0810 14:40:10.843659 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.843741 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:40:10.844012 14050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
I20260810 14:40:10.844980 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.844769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46632)
0810 14:40:10.844883 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.844929 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:40:10.845201 14110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
I20260810 14:40:10.871111 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 1
I20260810 14:40:10.872229 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.871823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46648)
0810 14:40:10.872063 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.872131 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 14:40:10.873034 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.872805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46656)
0810 14:40:10.872922 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.872977 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:40:10.873183 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
I20260810 14:40:10.894510 14110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 1
I20260810 14:40:10.895602 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.895238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46666)
0810 14:40:10.895417 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.895498 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:40:10.896512 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.896272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46676)
0810 14:40:10.896391 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.896447 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:40:10.896776 14110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
I20260810 14:40:10.948385 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 2
I20260810 14:40:10.949373 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.949052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46678)
0810 14:40:10.949228 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.949293 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 14:40:10.950184 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.949962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46682)
0810 14:40:10.950074 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.950130 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:40:10.950320 14023 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
W20260810 14:40:10.963713 14118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:10.964068 14118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:10.964134 14118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:10.964236 14118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:40:10.965000 14110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 2
I20260810 14:40:10.965921 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.965563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46690)
0810 14:40:10.965753 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.965825 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:40:10.966693 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:10.966455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46694)
0810 14:40:10.966567 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:10.966618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:40:10.966830 14110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:32917: Network error: Client connection negotiation failed: client connection to 127.21.43.124:32917: connect: Connection refused (error 111)
W20260810 14:40:10.969002 14118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:10.969054 14118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:10.969074 14118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:10.969091 14118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:10.969106 14118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:10.974577 14118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:43811,127.21.43.125:37123,127.21.43.124:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.43.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:10.975778 14118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:10.976905 14118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:10.985699 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:10.985756 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:10.985651 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:10.985883 14118 server_base.cc:1034] running on GCE node
I20260810 14:40:10.986614 14118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:10.987813 14118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:10.988981 14118 hybrid_clock.cc:648] HybridClock initialized: now 1786372810988964 us; error 31 us; skew 500 ppm
Aug 10 14:40:10 dist-test-slave-chsk krb5kdc[13929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:10.991881 14118 init.cc:377] Logged in from keytab as kudu/127.21.43.124@KRBTEST.COM (short username kudu)
I20260810 14:40:10.992830 14118 webserver.cc:460] Webserver started at http://127.21.43.124:39951/ using document root <none> and password file <none>
I20260810 14:40:10.993407 14118 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:10.993470 14118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:10.993741 14118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:10.995911 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data/instance:
uuid: "50abbe0e710d40a3b93a9b89922995d3"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "6520244eb1ffc786b3570cabc2438dbb"
server_key_iv: "6ed168c61f8ec51b9838dc192286e19e"
server_key_version: "encryptionkey@0"
I20260810 14:40:10.996497 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal/instance:
uuid: "50abbe0e710d40a3b93a9b89922995d3"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "6520244eb1ffc786b3570cabc2438dbb"
server_key_iv: "6ed168c61f8ec51b9838dc192286e19e"
server_key_version: "encryptionkey@0"
I20260810 14:40:11.000473 14118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:40:11.002921 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:11.004230 14118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:11.004381 14118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "50abbe0e710d40a3b93a9b89922995d3"
format_stamp: "Formatted at 2026-08-10 14:40:10 on dist-test-slave-chsk"
server_key: "6520244eb1ffc786b3570cabc2438dbb"
server_key_iv: "6ed168c61f8ec51b9838dc192286e19e"
server_key_version: "encryptionkey@0"
I20260810 14:40:11.004501 14118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:11.021447 14118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:11.024443 14118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:11.024658 14118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:11.032008 14118 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.124:32917
I20260810 14:40:11.032316 14196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.124:32917 every 8 connection(s)
I20260810 14:40:11.033022 14118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
I20260810 14:40:11.035796 14197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:11.039599 14197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:11.041290 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 14118
I20260810 14:40:11.041430 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372517778458-21677-0/minicluster-data/master-2/wal/instance
I20260810 14:40:11.041747 21677 external_mini_cluster.cc:1428] Setting key 4f0a0e649bd5edac997d2681e869a791
I20260810 14:40:11.049486 14023 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 3
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:40:11.055095 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.041590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.21.43.126:43811)
0810 14:40:11.041735 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.041738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.042829 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 14:40:11.043908 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.043911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.043962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.044066 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.045156 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.045941 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.047283 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.047578 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.047697 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.050855 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.050859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.050862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.050890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.052639 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.053293 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.053297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.053299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.053392 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.053831 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.053835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.053837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.054084 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.054188 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.054716 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.054839 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:40:11.056306 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.042463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.21.43.124:32917)
0810 14:40:11.042855 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.042866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.042906 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:40:11.043728 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.043748 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.043975 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.044154 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.045031 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.046258 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.047414 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.047920 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.048000 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.051225 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.051233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.051246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.051274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.053303 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.054381 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.054388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.054389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.054486 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.054821 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.054825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.054827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.055099 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.055374 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.055728 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.055882 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260810 14:40:11.056315 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.042096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:43811 (local address 127.0.0.1:44940)
0810 14:40:11.042665 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:11.042729 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:11.043715 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:11.044149 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:11.044156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:11.044634 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:11.044998 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.045019 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.046051 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.046053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.047120 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.047132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.047726 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.047729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.047868 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.048476 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:11.048520 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:11.050712 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:11.052759 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.052767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.052777 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.053160 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.053511 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.053514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.053516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.053701 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.054192 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:11.054198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:11.054454 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 14:40:11.054664 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 14:40:11.055075 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"spinlock_wait_cycles":1408,"thread_start_us":163,"threads_started":1}
I20260810 14:40:11.057982 14110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } attempt: 3
I20260810 14:40:11.058091 14197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:11.060422 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.049963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46708)
0810 14:40:11.050186 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:11.050201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:11.050277 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:11.050600 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:11.050603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:11.050800 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:11.050922 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.050930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.052001 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.052004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.052850 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.052857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.053280 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.053284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.053367 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.053808 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:11.053827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:11.055356 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:11.058733 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.058738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.058740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.058998 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.059361 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.059367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.059372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.059463 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.059941 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:11.059945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:11.060037 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 14:40:11.060118 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:40:11.060261 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260810 14:40:11.061064 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.050051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.21.43.124:32917)
0810 14:40:11.050363 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.050371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.050384 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:11.050416 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.050419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.050457 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.050552 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.051053 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.051854 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.052958 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.053166 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.053267 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.055612 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.055615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.055620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.055648 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.058589 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.059143 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.059148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.059150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.059254 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.059577 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.059582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.059587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.059822 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.060677 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.060759 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.060858 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":245,"thread_start_us":128,"threads_started":1}
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:40:11.064704 14023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: Bootstrap starting.
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:11.067905 14023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:11.068907 14023 log.cc:824] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:11.071132 14023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: No bootstrap required, opened a new log
I20260810 14:40:11.071306 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.058422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46720)
0810 14:40:11.058695 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:11.058713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:11.058796 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:11.059757 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:11.059760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:11.059956 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:11.060056 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.060064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.061025 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.061029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.061820 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.061830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.062489 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.062492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.062574 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.063296 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:11.063321 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:11.065871 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:11.068615 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.068618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.068620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.069278 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.069981 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.069987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.069992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.070108 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.070617 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:11.070620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:11.070741 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 14:40:11.070916 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 14:40:11.071069 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":31,"spinlock_wait_cycles":1408}
I20260810 14:40:11.071326 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.059233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.21.43.126:43811)
0810 14:40:11.059355 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.059359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.059384 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:40:11.059476 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.059480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.059525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.059602 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.060391 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.061518 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.062661 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.062962 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.063048 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.067588 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.067595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.067601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.067633 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.069354 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.069823 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.069826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.069828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.069908 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.070236 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.070239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.070240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.070461 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.070615 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.070963 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.071077 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:40:11.072206 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.058932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.21.43.124:32917)
0810 14:40:11.059531 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.059535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.059548 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:11.059605 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.059608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.059648 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.059736 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.060198 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.060904 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.061957 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.062347 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.066002 (+ 3655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.066706 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.066709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.066712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.066728 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.068487 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.069780 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.069786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.069790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.069872 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.070230 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.070233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.070235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.070508 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.071816 (+ 1308us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.071900 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.072028 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"spinlock_wait_cycles":7808}
I20260810 14:40:11.072360 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.059048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.21.43.125:37123)
0810 14:40:11.059196 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.059200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.059213 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:11.059249 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.059252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.059299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.059379 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.060265 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.061088 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.062197 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.062563 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.062648 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.065455 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.065459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.065461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.065494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.068957 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.070019 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.070024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.070029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.070108 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.070467 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.070470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.070472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.071665 (+ 1193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.071765 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.072097 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.072213 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":895,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":2432}
I20260810 14:40:11.073282 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.058563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:37123 (local address 127.0.0.1:57848)
0810 14:40:11.058814 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:11.058829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:11.058897 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:11.059845 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:11.059847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:11.060027 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:11.060118 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.060125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.061246 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.061249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.062063 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.062073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.062804 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.062808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.062894 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.063339 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:11.063355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:11.065316 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:11.069396 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.069402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.069407 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.069660 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.070230 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.070233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.070235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.070354 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.071819 (+ 1465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:11.071822 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:11.071907 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 14:40:11.072969 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0810 14:40:11.073120 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"spinlock_wait_cycles":1408}
I20260810 14:40:11.075429 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: Bootstrap starting.
I20260810 14:40:11.075883 14023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.076125 14023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14010e5fec56475ab4334aee036e8f54, State: Initialized, Role: FOLLOWER
I20260810 14:40:11.076470 14023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.077087 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.077410 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.077926 14023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:11.078387 14110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: Neither blocks nor log segments found. Creating new log.
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:40:11.079569 14110 log.cc:824] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:11.079628 14197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:11.081944 14110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: No bootstrap required, opened a new log
Aug 10 14:40:11 dist-test-slave-chsk krb5kdc[13929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372810, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:40:11.085886 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.073029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.21.43.125:37123)
0810 14:40:11.073134 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.073139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.073444 (+ 305us) server_negotiation.cc:407] Connection header received
0810 14:40:11.073480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.073483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.073525 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.073591 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.074309 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.074821 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.075842 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.076070 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.076188 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.079471 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.079476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.079480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.079511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.082567 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.083083 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.083087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.083089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.083233 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.083545 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.083549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.083551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.083799 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.083970 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.085572 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.085726 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 14:40:11.086913 14216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:11.087008 14216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:11.089327 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:11.085498 13946 master_runner.cc:186] Error getting master registration for 127.21.43.125:37123: OK, instance_id { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" instance_seqno: 1786372810802709 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:40:11.089637 13946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:40:11.089823 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.080550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:32917 (local address 127.0.0.1:46734)
0810 14:40:11.080693 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:11.080708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:11.080798 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:11.081143 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:11.081146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:11.081357 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:11.081478 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.081486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.082451 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.082455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.083102 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:11.083112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.083800 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.083803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:11.083890 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.084371 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:11.084387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:11.086023 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:11.088312 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.088315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.088317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.088558 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.088823 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:11.088826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:11.088828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:11.088897 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:11.089335 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:11.089338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:11.089426 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 14:40:11.089526 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:40:11.089654 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 14:40:11.090346 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.080802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.21.43.124:32917)
0810 14:40:11.080911 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.080915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.080929 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:11.080966 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.080969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.081010 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.081095 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.081611 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.082325 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.083263 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.083605 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.083700 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.086404 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.086408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.086410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.086438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.088193 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.088651 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.088655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.088656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.088733 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.088979 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.088982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.088985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.089204 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.089300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.090077 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.090205 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:40:11.091238 13994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "14010e5fec56475ab4334aee036e8f54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44956
I20260810 14:40:11.091562 13994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:40:11.091703 13994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:11.091615 14110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.091861 14110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d7c97659fa45668dbd4025e2befdac, State: Initialized, Role: FOLLOWER
I20260810 14:40:11.092221 14110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.093062 13994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.093596 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: Bootstrap starting.
I20260810 14:40:11.093652 14110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:11.093995 13994 leader_election.cc:288] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [CANDIDATE]: Term 1 election: Requested vote from peers 18d7c97659fa45668dbd4025e2befdac (127.21.43.125:37123), 50abbe0e710d40a3b93a9b89922995d3 (127.21.43.124:32917)
I20260810 14:40:11.094795 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14010e5fec56475ab4334aee036e8f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18d7c97659fa45668dbd4025e2befdac"
I20260810 14:40:11.095024 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:11.096369 14081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14010e5fec56475ab4334aee036e8f54 in term 1.
I20260810 14:40:11.096433 14197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:11.096837 13966 leader_election.cc:302] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14010e5fec56475ab4334aee036e8f54, 18d7c97659fa45668dbd4025e2befdac; no voters:
I20260810 14:40:11.097131 14204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:11.097430 14204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [term 1 LEADER]: Becoming Leader. State: Replica: 14010e5fec56475ab4334aee036e8f54, State: Running, Role: LEADER
I20260810 14:40:11.097838 14204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.099815 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14010e5fec56475ab4334aee036e8f54. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.099946 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:11.100574 14197 log.cc:824] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:11.103066 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.103132 14237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:11.103210 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.104076 14237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:11.108274 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: No bootstrap required, opened a new log
I20260810 14:40:11.104832 14030 master_runner.cc:186] Error getting master registration for 127.21.43.124:32917: OK, instance_id { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" instance_seqno: 1786372811029911 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:40:11.108614 14030 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 14:40:11.110833 14235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:11.111083 14235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:11.115814 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.103725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.21.43.126:43811)
0810 14:40:11.103852 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.103859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.105406 (+ 1547us) server_negotiation.cc:407] Connection header received
0810 14:40:11.105586 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.105589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.105637 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.105686 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.106770 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.107677 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.108960 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.109297 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.109380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.111452 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.111456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.111459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.111489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.113403 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.114049 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.114052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.114055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.114143 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.114446 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.114449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.114451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.114682 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.114787 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.115520 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.115643 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:11.116909 14197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.117235 14197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50abbe0e710d40a3b93a9b89922995d3, State: Initialized, Role: FOLLOWER
I20260810 14:40:11.117609 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } }
I20260810 14:40:11.119462 14197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:11.119515 14166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14010e5fec56475ab4334aee036e8f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50abbe0e710d40a3b93a9b89922995d3"
I20260810 14:40:11.119823 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:11.121731 14166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14010e5fec56475ab4334aee036e8f54 in term 1.
I20260810 14:40:11.123267 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.104495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.21.43.125:37123)
0810 14:40:11.104629 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.104633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.111707 (+ 7074us) server_negotiation.cc:407] Connection header received
0810 14:40:11.112054 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.112058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.112118 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.112183 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.113827 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.114557 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.115853 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.116163 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.116241 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.118770 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.118774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.118777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.118810 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.121067 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.121673 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.121677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.121679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.121783 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.122212 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.122216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.122217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.122454 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.122559 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.122950 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.123081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:40:11.125520 14166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [term 1 FOLLOWER]: Refusing update from remote peer 14010e5fec56475ab4334aee036e8f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:11.126183 14204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:11.130877 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14010e5fec56475ab4334aee036e8f54. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.131176 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.131424 14246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.131585 14246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.131578 14081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [term 1 FOLLOWER]: Refusing update from remote peer 14010e5fec56475ab4334aee036e8f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:11.132192 14204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:11.136679 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14010e5fec56475ab4334aee036e8f54. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.136811 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.139199 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:11.105125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.21.43.124:32917)
0810 14:40:11.105237 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.105241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.112380 (+ 7139us) server_negotiation.cc:407] Connection header received
0810 14:40:11.113000 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.113005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.113097 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.113185 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.122727 (+ 9542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.131187 (+ 8460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.132313 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:11.132631 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:11.132747 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:11.134778 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:11.134782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:11.134786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:11.134815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:11.136806 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.137380 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.137384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.137386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.137478 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:11.137805 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:11.137808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:11.137809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:11.138024 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:11.138149 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:11.138858 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 14:40:11.138980 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7674,"server-negotiator.queue_time_us":46}
I20260810 14:40:11.142740 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.142879 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:11.142966 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.143033 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:11.144555 14237 catalog_manager.cc:1347] Generated new cluster ID: 0dd2d902f50d48b7a7b7fabb18e7fc7e
I20260810 14:40:11.144623 14237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:11.144836 14246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.144944 14246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.146013 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14010e5fec56475ab4334aee036e8f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14010e5fec56475ab4334aee036e8f54" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 43811 } } peers { permanent_uuid: "18d7c97659fa45668dbd4025e2befdac" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 37123 } } peers { permanent_uuid: "50abbe0e710d40a3b93a9b89922995d3" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 32917 } } }
I20260810 14:40:11.146135 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:11.147774 14256 mvcc.cc:205] Tried to move back new op lower bound from 7316983034362064896 to 7316983034260594688. Current Snapshot: MvccSnapshot[applied={T|T < 7316983034362064896}]
I20260810 14:40:11.147955 14245 mvcc.cc:205] Tried to move back new op lower bound from 7316983034362064896 to 7316983034260594688. Current Snapshot: MvccSnapshot[applied={T|T < 7316983034362064896 or (T in {7316983034362064896})}]
I20260810 14:40:11.148437 14266 mvcc.cc:205] Tried to move back new op lower bound from 7316983034362064896 to 7316983034260594688. Current Snapshot: MvccSnapshot[applied={T|T < 7316983034362064896}]
W20260810 14:40:11.151813 14265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:11.151903 14265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:11.157501 14237 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:11.158818 14237 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:11.174098 14237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14010e5fec56475ab4334aee036e8f54: Generated new TSK 0
I20260810 14:40:11.175065 14237 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:40:11.211387 14026 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.208059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.21.43.126:43811)
0810 14:40:11.208216 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.208221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.210143 (+ 1922us) server_negotiation.cc:407] Connection header received
0810 14:40:11.210358 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.210363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.210418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.210472 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.211227 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44972: BlockingRecv error: recv got EOF from 127.0.0.1:44972 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 14:40:11.213009 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.208914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.21.43.125:37123)
0810 14:40:11.209026 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.209030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.212074 (+ 3044us) server_negotiation.cc:407] Connection header received
0810 14:40:11.212254 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.212258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.212307 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.212357 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.212879 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57860: BlockingRecv error: recv got EOF from 127.0.0.1:57860 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 14:40:11.214712 14201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.209549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.21.43.124:32917)
0810 14:40:11.209666 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.209670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.213542 (+ 3872us) server_negotiation.cc:407] Connection header received
0810 14:40:11.213935 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.213939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.213991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.214053 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.214586 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46748: BlockingRecv error: recv got EOF from 127.0.0.1:46748 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 14:40:11.219475 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.217804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.21.43.125:37123)
0810 14:40:11.217912 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.217917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.218449 (+ 532us) server_negotiation.cc:407] Connection header received
0810 14:40:11.218679 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.218682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.218760 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.218811 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.219368 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57864: BlockingRecv error: recv got EOF from 127.0.0.1:57864 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 14:40:11.224809 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.223452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.21.43.125:37123)
0810 14:40:11.223583 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.223588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.223910 (+ 322us) server_negotiation.cc:407] Connection header received
0810 14:40:11.224105 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.224108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.224160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.224211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.224705 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57872: BlockingRecv error: recv got EOF from 127.0.0.1:57872 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 14:40:11.230872 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.229291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.21.43.125:37123)
0810 14:40:11.229397 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.229402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.229939 (+ 537us) server_negotiation.cc:407] Connection header received
0810 14:40:11.230143 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.230147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.230199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.230250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.230766 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57878: BlockingRecv error: recv got EOF from 127.0.0.1:57878 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:40:11.238811 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.237157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.21.43.125:37123)
0810 14:40:11.237312 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.237317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.237859 (+ 542us) server_negotiation.cc:407] Connection header received
0810 14:40:11.238079 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.238082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.238132 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.238180 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.238699 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57888: BlockingRecv error: recv got EOF from 127.0.0.1:57888 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 14:40:11.251343 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.249480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.21.43.125:37123)
0810 14:40:11.249614 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.249618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.250245 (+ 627us) server_negotiation.cc:407] Connection header received
0810 14:40:11.250460 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.250464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.250521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.250576 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.251213 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57896: BlockingRecv error: recv got EOF from 127.0.0.1:57896 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:40:11.272315 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.270188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.21.43.125:37123)
0810 14:40:11.270321 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.270326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.270987 (+ 661us) server_negotiation.cc:407] Connection header received
0810 14:40:11.271270 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.271275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.271336 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.271398 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.272162 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57902: BlockingRecv error: recv got EOF from 127.0.0.1:57902 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 14:40:11.309548 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.307339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.21.43.125:37123)
0810 14:40:11.307483 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.307496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.308295 (+ 799us) server_negotiation.cc:407] Connection header received
0810 14:40:11.308525 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.308529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.308605 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.308664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.309384 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57916: BlockingRecv error: recv got EOF from 127.0.0.1:57916 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 14:40:11.377837 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.376506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.21.43.125:37123)
0810 14:40:11.376661 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.376666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.376825 (+ 159us) server_negotiation.cc:407] Connection header received
0810 14:40:11.376967 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.376973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.377036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.377158 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.377703 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57932: BlockingRecv error: recv got EOF from 127.0.0.1:57932 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 14:40:11.509295 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.508123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.21.43.125:37123)
0810 14:40:11.508302 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.508306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.508408 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:40:11.508560 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.508566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.508643 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.508719 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.509164 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57948: BlockingRecv error: recv got EOF from 127.0.0.1:57948 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 14:40:11.768711 14116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:11.767677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.21.43.125:37123)
0810 14:40:11.767846 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:11.767850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:11.767868 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:40:11.768038 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:11.768040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:11.768092 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:11.768179 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:11.768607 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57964: BlockingRecv error: recv got EOF from 127.0.0.1:57964 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 14:40:12.112645 14235 catalog_manager.cc:1259] Loaded cluster ID: 0dd2d902f50d48b7a7b7fabb18e7fc7e
I20260810 14:40:12.112771 14235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: loading cluster ID for follower catalog manager: success
I20260810 14:40:12.114817 14235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: acquiring CA information for follower catalog manager: success
I20260810 14:40:12.116422 14235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18d7c97659fa45668dbd4025e2befdac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:40:12.153326 14265 catalog_manager.cc:1259] Loaded cluster ID: 0dd2d902f50d48b7a7b7fabb18e7fc7e
I20260810 14:40:12.153411 14265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: loading cluster ID for follower catalog manager: success
I20260810 14:40:12.155273 14265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: acquiring CA information for follower catalog manager: success
I20260810 14:40:12.156565 14265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50abbe0e710d40a3b93a9b89922995d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:40:12.286804 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:12.283007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.21.43.125:37123)
0810 14:40:12.283332 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:12.283336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:12.283353 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:12.283442 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:12.283444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:12.283580 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:12.283680 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:12.283898 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.284430 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.285234 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.285448 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.285524 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:12.285905 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:12.286361 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:12.286467 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:12.286524 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 14:40:12.286588 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260810 14:40:12.294101 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:12.290194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.21.43.124:32917)
0810 14:40:12.290460 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:12.290464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:12.290478 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:12.290680 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:12.290683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:12.290801 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:12.290927 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:12.291176 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.291726 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.292561 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.292810 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.292913 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:12.293236 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:12.293672 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:12.293802 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:12.293866 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 14:40:12.293926 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260810 14:40:12.301923 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:12.298075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.21.43.124:32917)
0810 14:40:12.298414 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:12.298418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:12.298430 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:40:12.298471 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:12.298476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:12.298520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:12.298599 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:12.298859 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.299530 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.300711 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.300983 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.301067 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:12.301380 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:12.301443 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:12.301524 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:12.301704 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 14:40:12.301774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260810 14:40:12.302160 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:12.297301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.21.43.126:43811)
0810 14:40:12.297608 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:12.297613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:12.297632 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:40:12.297739 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:12.297742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:12.297927 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:12.298091 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:12.298319 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.299259 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.300298 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.300547 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.300663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:12.301128 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:12.301491 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:12.301616 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:12.301898 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 14:40:12.301976 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260810 14:40:12.302531 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:12.297622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.21.43.125:37123)
0810 14:40:12.297761 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:12.297764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:12.298139 (+ 375us) server_negotiation.cc:407] Connection header received
0810 14:40:12.298349 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:12.298352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:12.298420 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:12.298603 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:12.299018 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.299797 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.301036 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:12.301328 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:12.301406 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:12.301821 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:12.301899 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:12.301946 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:12.302349 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 14:40:12.302412 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:40:12.305523 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 13946
I20260810 14:40:12.315392 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 14030
I20260810 14:40:12.324928 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 14118
2026-08-10T14:40:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3172 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:40:12 dist-test-slave-chsk krb5kdc[14366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:40:12 dist-test-slave-chsk krb5kdc[14366](info): set up 2 sockets
Aug 10 14:40:12 dist-test-slave-chsk krb5kdc[14366](info): commencing operation
krb5kdc: starting...
W20260810 14:40:13.425751 21677 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/test-user.keytab.
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.43.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
Entry for principal kudu/127.21.43.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab.
I20260810 14:40:13.558204 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39803
--webserver_interface=127.21.43.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:40:13.673194 14381 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:13.673508 14381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:13.673542 14381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:13.673583 14381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:13.677613 14381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:13.677673 14381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:13.677693 14381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:13.677711 14381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:13.677726 14381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:13.683090 14381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.126.keytab
--principal=kudu/127.21.43.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.126:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.43.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:13.684231 14381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:13.685252 14381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:13.693928 14386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:13.693957 14387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:13.693939 14389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:13.694342 14381 server_base.cc:1034] running on GCE node
I20260810 14:40:13.694976 14381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:13.696197 14381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:13.697397 14381 hybrid_clock.cc:648] HybridClock initialized: now 1786372813697364 us; error 51 us; skew 500 ppm
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:13.700350 14381 init.cc:377] Logged in from keytab as kudu/127.21.43.126@KRBTEST.COM (short username kudu)
I20260810 14:40:13.701313 14381 webserver.cc:460] Webserver started at http://127.21.43.126:45377/ using document root <none> and password file <none>
I20260810 14:40:13.701917 14381 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:13.702031 14381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:13.702291 14381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:13.704475 14381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data/instance:
uuid: "ee4854e16a894b11960b6483a1fb84f2"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "dc8ee17e44848bbb073919d65fa78775"
server_key_iv: "af8faa25194511e991a7a766dca1a1fd"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.705080 14381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance:
uuid: "ee4854e16a894b11960b6483a1fb84f2"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "dc8ee17e44848bbb073919d65fa78775"
server_key_iv: "af8faa25194511e991a7a766dca1a1fd"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.708835 14381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:40:13.711127 14396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:13.712373 14381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:40:13.712523 14381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal
uuid: "ee4854e16a894b11960b6483a1fb84f2"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "dc8ee17e44848bbb073919d65fa78775"
server_key_iv: "af8faa25194511e991a7a766dca1a1fd"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.712639 14381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:13.729643 14381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:13.733306 14381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:13.733522 14381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:13.741396 14381 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.126:39803
I20260810 14:40:13.741686 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.126:39803 every 8 connection(s)
I20260810 14:40:13.742461 14381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/data/info.pb
I20260810 14:40:13.745249 14457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:13.745602 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 14381
I20260810 14:40:13.745745 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-0/wal/instance
I20260810 14:40:13.746060 21677 external_mini_cluster.cc:1428] Setting key f6a4cb546eaea1912d1333fc758dad5f
I20260810 14:40:13.749261 14457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.125@KRBTEST.COM" created.
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:13.765555 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.752025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39803 (local address 127.0.0.1:51138)
0810 14:40:13.752505 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.752556 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:13.753383 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:13.754188 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:13.754192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:13.754503 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:13.754768 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.754780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.756453 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.756455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.757173 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.757181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.757914 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.757916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.758007 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.758663 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:13.758679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:13.760525 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:13.762405 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.762416 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.762425 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.762653 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.763060 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.763063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.763065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.763244 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.763844 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:13.763849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:13.763988 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 14:40:13.764166 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 14:40:13.764576 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":121,"thread_start_us":146,"threads_started":1}
I20260810 14:40:13.765555 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.752026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.21.43.126:39803)
0810 14:40:13.752619 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:13.752634 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:13.752686 (+ 52us) server_negotiation.cc:407] Connection header received
0810 14:40:13.753698 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:13.753706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:13.753972 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:13.754199 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:13.754935 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.756336 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.757313 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.757776 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.757904 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.760694 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:13.760700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:13.760710 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:13.760730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:13.762292 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.762772 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.762780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.762789 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.762942 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.763344 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.763347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.763348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.763719 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:13.763859 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:13.764229 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 14:40:13.764528 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":459,"spinlock_wait_cycles":896,"thread_start_us":162,"threads_started":1}
I20260810 14:40:13.768353 14457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:13.769155 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.768868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33916)
0810 14:40:13.769005 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.769088 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 14:40:13.769425 14403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
I20260810 14:40:13.770291 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.770054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33918)
0810 14:40:13.770197 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.770239 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:40:13.770457 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:36197: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
Entry for principal kudu/127.21.43.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab.
I20260810 14:40:13.774336 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:36197
--webserver_interface=127.21.43.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:13.819716 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } attempt: 1
I20260810 14:40:13.820657 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.820318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33930)
0810 14:40:13.820496 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.820572 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 14:40:13.821434 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.821204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33936)
0810 14:40:13.821322 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.821376 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 14:40:13.821578 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:36197: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
W20260810 14:40:13.887392 14465 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:13.887774 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:13.887845 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:13.887928 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:40:13.889777 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } attempt: 2
I20260810 14:40:13.890669 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.890351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33950)
0810 14:40:13.890514 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.890584 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 14:40:13.891535 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.891298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33960)
0810 14:40:13.891413 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.891466 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:40:13.891721 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.125:36197: Network error: Client connection negotiation failed: client connection to 127.21.43.125:36197: connect: Connection refused (error 111)
W20260810 14:40:13.892675 14465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:13.892730 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:13.892771 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:13.892808 14465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:13.892845 14465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:13.898406 14465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.125.keytab
--principal=kudu/127.21.43.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.125:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.43.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:13.899564 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:13.900609 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:40:13.909318 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:13.909325 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:13.909377 14465 server_base.cc:1034] running on GCE node
W20260810 14:40:13.909325 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:13.910192 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:13.911408 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:13.912575 14465 hybrid_clock.cc:648] HybridClock initialized: now 1786372813912558 us; error 27 us; skew 500 ppm
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:13.915375 14465 init.cc:377] Logged in from keytab as kudu/127.21.43.125@KRBTEST.COM (short username kudu)
I20260810 14:40:13.916365 14465 webserver.cc:460] Webserver started at http://127.21.43.125:36501/ using document root <none> and password file <none>
I20260810 14:40:13.916932 14465 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:13.917035 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:13.917279 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:13.919437 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data/instance:
uuid: "3ed7fbb88b904957821772410ab8be8a"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "62e7dceede45146fb37bea8b2f50355a"
server_key_iv: "6bf8095b581a1931f32bf9c7a106d50f"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.920022 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal/instance:
uuid: "3ed7fbb88b904957821772410ab8be8a"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "62e7dceede45146fb37bea8b2f50355a"
server_key_iv: "6bf8095b581a1931f32bf9c7a106d50f"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.923840 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:40:13.926308 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:13.927616 14465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 14:40:13.927772 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal
uuid: "3ed7fbb88b904957821772410ab8be8a"
format_stamp: "Formatted at 2026-08-10 14:40:13 on dist-test-slave-chsk"
server_key: "62e7dceede45146fb37bea8b2f50355a"
server_key_iv: "6bf8095b581a1931f32bf9c7a106d50f"
server_key_version: "encryptionkey@0"
I20260810 14:40:13.927884 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:13.944309 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:13.947484 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:13.947703 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:13.954993 14465 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.125:36197
I20260810 14:40:13.955371 14537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.125:36197 every 8 connection(s)
I20260810 14:40:13.956045 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/data/info.pb
I20260810 14:40:13.958894 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:13.961373 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 14465
I20260810 14:40:13.961529 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-1/wal/instance
I20260810 14:40:13.961843 21677 external_mini_cluster.cc:1428] Setting key 48cdf6c4f46f3e459951c0a1057a1f70
I20260810 14:40:13.963914 14538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.43.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.43.124@KRBTEST.COM" created.
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:13.980786 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.966087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.21.43.126:39803)
0810 14:40:13.966280 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:13.966287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:13.967240 (+ 953us) server_negotiation.cc:407] Connection header received
0810 14:40:13.968782 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:13.968785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:13.968842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:13.968923 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:13.970616 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.971269 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.972732 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.973039 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.973149 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.976334 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:13.976337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:13.976340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:13.976369 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:13.978163 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.978736 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.978742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.978746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.978844 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.979308 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.979314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.979319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.979605 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:13.979732 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:13.980333 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 14:40:13.980501 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:40:13.981564 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.966429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39803 (local address 127.0.0.1:51150)
0810 14:40:13.967024 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.967090 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:13.968471 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:13.969159 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:13.969171 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:13.969711 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:13.970434 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.970459 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.971413 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.971420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.972592 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.972603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.973224 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.973228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.973386 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.974133 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:13.974151 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:13.976113 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:13.978312 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.978318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.978331 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.978628 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.978970 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.978973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.978974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.979135 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.979740 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:13.979745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:13.980040 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 14:40:13.980204 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 14:40:13.980583 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":144,"threads_started":1}
I20260810 14:40:13.982882 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } attempt: 3
I20260810 14:40:13.983299 14538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
Aug 10 14:40:13 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
Entry for principal kudu/127.21.43.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab.
I20260810 14:40:13.994722 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.983340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33966)
0810 14:40:13.983736 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.983754 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:13.983838 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:13.984726 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:13.984732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:13.984925 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:13.985055 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.985063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.986348 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.986351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.987462 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.987472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.988162 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.988165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.988289 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.988795 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:13.988806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:13.990362 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:13.992937 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.992942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.992945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.993168 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.993549 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.993560 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.993665 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.994190 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:13.994194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:13.994299 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 14:40:13.994440 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 14:40:13.994583 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260810 14:40:13.994702 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.983853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33976)
0810 14:40:13.983998 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.984012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:13.984082 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:13.984776 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:13.984779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:13.984970 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:13.985141 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.985149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.986211 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.986216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.986744 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:13.986751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.987571 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.987576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:13.987671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.988210 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:13.988228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:13.989838 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:13.992934 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.992937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.992939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.993140 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.993482 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:13.993486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:13.993579 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:13.994134 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:13.994139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:13.994233 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 14:40:13.994343 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:40:13.994500 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"spinlock_wait_cycles":10624}
I20260810 14:40:13.994710 21677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA76VjH/build/debug/bin/kudu
/tmp/dist-test-taskA76VjH/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:37439
--webserver_interface=127.21.43.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.43.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:40:13.995393 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.983574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.21.43.125:36197)
0810 14:40:13.984157 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:13.984169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:13.984196 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:40:13.984294 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:13.984301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:13.984551 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:13.984694 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:13.985219 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.986130 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.987599 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.988023 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.988145 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.990521 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:13.990524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:13.990526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:13.990559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:13.992767 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993309 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.993314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.993316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.993423 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993803 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.993809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.993814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.994048 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:13.995040 (+ 992us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:13.995102 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 14:40:13.995268 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":298,"server-negotiator.queue_time_us":478,"spinlock_wait_cycles":17408,"thread_start_us":156,"threads_started":1}
I20260810 14:40:13.995589 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.984070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.21.43.125:36197)
0810 14:40:13.984382 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:13.984390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:13.984401 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:40:13.984427 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:13.984429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:13.984551 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:13.984694 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:13.985345 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.986078 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.986861 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:13.987400 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:13.987539 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:13.990002 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:13.990012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:13.990055 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:13.990087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:13.992793 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993264 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.993271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.993272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.993384 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:13.993671 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:13.993676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:13.993678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:13.994001 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:13.994166 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:13.994462 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 14:40:13.994637 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":249,"spinlock_wait_cycles":15488,"thread_start_us":116,"threads_started":1}
I20260810 14:40:13.997498 14457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:13.997624 14538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:13.998363 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.998024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58612)
0810 14:40:13.998184 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.998255 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 14:40:13.998421 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.998070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58626)
0810 14:40:13.998250 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.998337 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 14:40:13.998608 14484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:13.999233 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.998864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58638)
0810 14:40:13.999072 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.999124 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260810 14:40:13.999408 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:13.999665 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:13.999371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58652)
0810 14:40:13.999529 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:13.999584 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 14:40:13.999820 14538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:14.030606 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 1
I20260810 14:40:14.031720 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.031367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58668)
0810 14:40:14.031522 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.031601 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 14:40:14.032547 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.032333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58676)
0810 14:40:14.032445 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.032493 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:40:14.032687 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:14.049109 14538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 1
I20260810 14:40:14.050102 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.049750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58686)
0810 14:40:14.049930 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.050007 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 14:40:14.050897 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.050657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58688)
0810 14:40:14.050772 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.050824 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:40:14.051076 14538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:14.107862 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 2
I20260810 14:40:14.108803 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.108484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58700)
0810 14:40:14.108639 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.108715 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 14:40:14.109583 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.109376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58712)
0810 14:40:14.109474 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.109520 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:40:14.109779 14457 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
W20260810 14:40:14.113545 14546 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:40:14.113844 14546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:40:14.113917 14546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:40:14.114018 14546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:40:14.118721 14546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:40:14.118784 14546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:40:14.118804 14546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:40:14.118822 14546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:40:14.118871 14546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:40:14.119289 14538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 2
I20260810 14:40:14.120146 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.119800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58724)
0810 14:40:14.119967 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.120044 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 14:40:14.120946 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.120711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58734)
0810 14:40:14.120826 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.120879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:40:14.121155 14538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.43.124:37439: Network error: Client connection negotiation failed: client connection to 127.21.43.124:37439: connect: Connection refused (error 111)
I20260810 14:40:14.125819 14546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.43.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.43.126:39803,127.21.43.125:36197,127.21.43.124:37439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/krb5kdc/kudu_127.21.43.124.keytab
--principal=kudu/127.21.43.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.43.124:37439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.43.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 13:59:06 UTC on ccycloud.quasar-wwztxk.root.comops.site
build id 4577958
I20260810 14:40:14.127202 14546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:40:14.128368 14546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:40:14.137640 14546 server_base.cc:1034] running on GCE node
W20260810 14:40:14.137645 14555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:14.137600 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:40:14.137598 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:40:14.138414 14546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:40:14.139708 14546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:40:14.140888 14546 hybrid_clock.cc:648] HybridClock initialized: now 1786372814140854 us; error 45 us; skew 500 ppm
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:40:14.143949 14546 init.cc:377] Logged in from keytab as kudu/127.21.43.124@KRBTEST.COM (short username kudu)
I20260810 14:40:14.144915 14546 webserver.cc:460] Webserver started at http://127.21.43.124:44293/ using document root <none> and password file <none>
I20260810 14:40:14.145505 14546 fs_manager.cc:359] Metadata directory not provided
I20260810 14:40:14.145571 14546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:40:14.145767 14546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:40:14.148205 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data/instance:
uuid: "0884fba5cc9a4664b7d01d8c431d9e38"
format_stamp: "Formatted at 2026-08-10 14:40:14 on dist-test-slave-chsk"
server_key: "c1b64a04a33cc9e9a70a547b4763a38e"
server_key_iv: "3c33f909324b90cabf6075c0c1c41584"
server_key_version: "encryptionkey@0"
I20260810 14:40:14.148826 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal/instance:
uuid: "0884fba5cc9a4664b7d01d8c431d9e38"
format_stamp: "Formatted at 2026-08-10 14:40:14 on dist-test-slave-chsk"
server_key: "c1b64a04a33cc9e9a70a547b4763a38e"
server_key_iv: "3c33f909324b90cabf6075c0c1c41584"
server_key_version: "encryptionkey@0"
I20260810 14:40:14.153571 14546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:40:14.156329 14564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:40:14.157708 14546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:40:14.157831 14546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data,/tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal
uuid: "0884fba5cc9a4664b7d01d8c431d9e38"
format_stamp: "Formatted at 2026-08-10 14:40:14 on dist-test-slave-chsk"
server_key: "c1b64a04a33cc9e9a70a547b4763a38e"
server_key_iv: "3c33f909324b90cabf6075c0c1c41584"
server_key_version: "encryptionkey@0"
I20260810 14:40:14.157938 14546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:40:14.178566 14546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:40:14.182308 14546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:40:14.182497 14546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:40:14.190188 14546 rpc_server.cc:307] RPC server started. Bound to: 127.21.43.124:37439
I20260810 14:40:14.190487 14622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.43.124:37439 every 8 connection(s)
I20260810 14:40:14.191346 14546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/data/info.pb
I20260810 14:40:14.193930 21677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA76VjH/build/debug/bin/kudu as pid 14546
I20260810 14:40:14.194118 21677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA76VjH/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372517778458-21677-0/minicluster-data/master-2/wal/instance
I20260810 14:40:14.194329 14623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:40:14.194475 21677 external_mini_cluster.cc:1428] Setting key eb9c602e8916e3c38d207e516d4989a4
I20260810 14:40:14.199476 14623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
I20260810 14:40:14.208942 14457 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 3
I20260810 14:40:14.209975 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.195931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.21.43.124:37439)
0810 14:40:14.196367 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.196384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.196422 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:40:14.197098 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.197116 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.197468 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.197758 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.198755 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.199888 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.201093 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.201708 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.201870 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.204548 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.204554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.204563 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.204589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.206487 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.207044 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.207050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.207057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.207248 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.207638 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.207642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.207644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.208040 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.208248 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.208567 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.208852 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260810 14:40:14.211544 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.201474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.21.43.126:39803)
0810 14:40:14.201638 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.201643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.201971 (+ 328us) server_negotiation.cc:407] Connection header received
0810 14:40:14.202202 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.202205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.202258 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.202349 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.203021 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.203574 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.204527 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.204739 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.204818 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.207479 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.207484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.207489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.207517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.209294 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.209861 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.209864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.209866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.209955 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.210533 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.210536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.210537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.210767 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.210865 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.211276 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.211411 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:40:14.212070 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.201412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.126:39803 (local address 127.0.0.1:51164)
0810 14:40:14.201854 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.201903 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:14.202087 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:14.202398 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:14.202403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:14.202763 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:14.202888 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.202894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.203695 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.203698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.204375 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.204385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.204859 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.204865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.204962 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.205422 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:14.205444 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:14.207332 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:14.209438 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.209447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.209453 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.209755 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.210329 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.210332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.210334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.210426 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.210869 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:14.210876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:14.210987 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:40:14.211710 (+ 723us) client_negotiation.cc:240] Negotiation successful
0810 14:40:14.211906 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":153,"threads_started":1}
I20260810 14:40:14.212335 14538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } attempt: 3
I20260810 14:40:14.214078 14623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.126@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.125@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.126@KRBTEST.COM
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:40:14.221011 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.209410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58738)
0810 14:40:14.209593 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.209608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:14.209681 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:14.210078 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:14.210081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:14.210262 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:14.210367 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.210374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.211412 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.211418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.212104 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.212114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.213331 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.213335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.213426 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.213960 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:14.213975 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:14.216168 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:14.218755 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.218759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.218761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.219385 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.219747 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.219750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.219752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.219860 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.220402 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:14.220405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:14.220519 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 14:40:14.220629 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:40:14.220808 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260810 14:40:14.221881 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.209507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.21.43.124:37439)
0810 14:40:14.209854 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.209864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.209875 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:40:14.209904 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.209907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.209952 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.210054 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.210487 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.211299 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.212863 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.213188 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.216301 (+ 3113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.216404 (+ 103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.216407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.216410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.216425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.218634 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.219531 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.219534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.219544 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.219631 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.219985 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.219990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.219994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.220225 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.220342 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.221501 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.221660 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260810 14:40:14.224655 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.213349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.21.43.126:39803)
0810 14:40:14.213490 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.213495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.213508 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:14.213540 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.213543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.213591 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.213665 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.214527 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.215335 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.216520 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.216873 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.216968 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.220468 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.220472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.220475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.220496 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.222259 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.222804 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.222807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.222810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.222906 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223255 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.223258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.223259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.223484 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.223718 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.224372 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.224498 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":106880}
I20260810 14:40:14.225179 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.214698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.21.43.125:36197)
0810 14:40:14.214813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.214819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.214836 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:14.214876 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.214879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.214915 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.215016 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.215955 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.216683 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.217906 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.218321 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.218412 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.220817 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.220821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.220825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.220854 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.222697 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223389 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.223392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.223394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.223500 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223834 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.223837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.223839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.224186 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.224341 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.224859 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.224992 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:14.225600 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.212997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58748)
0810 14:40:14.213314 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.213330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:14.213412 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:14.213889 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:14.213892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:14.214093 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:14.214236 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.214244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.215444 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.215464 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.216331 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.216343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.217071 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.217077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.217163 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.217661 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:14.217672 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:14.219871 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:14.223401 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.223406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.223718 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.224534 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.224540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.224543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.224642 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.225130 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:14.225133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:14.225228 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 14:40:14.225315 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:40:14.225454 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260810 14:40:14.225833 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.214539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.125:36197 (local address 127.0.0.1:33986)
0810 14:40:14.214704 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.214719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:14.214791 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:14.215038 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:14.215041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:14.215254 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:14.215806 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.215818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.216829 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.216833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.217755 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.217769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.218581 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.218585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.218699 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.219212 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:14.219227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:14.220650 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:14.222826 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.222831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.222836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.223263 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.223627 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.223636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.223719 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.224555 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:14.224557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:14.224660 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 14:40:14.225409 (+ 749us) client_negotiation.cc:240] Negotiation successful
0810 14:40:14.225544 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":1}
I20260810 14:40:14.225983 14457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: Bootstrap starting.
I20260810 14:40:14.226123 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.213533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.21.43.124:37439)
0810 14:40:14.213653 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.213658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.213669 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:40:14.213700 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.213702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.213742 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.214407 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.214511 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.215302 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.216606 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.216942 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.217381 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.220411 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.220414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.220417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.220440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.223290 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.223834 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.223837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.223839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.224425 (+ 586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.224774 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.224780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.224785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.224984 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.225785 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.225865 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.225989 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:40:14.226657 14623 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:40:14.229064 14538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: Bootstrap starting.
I20260810 14:40:14.230495 14457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:14.231494 14538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:14.231724 14457 log.cc:824] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:14.232286 14538 log.cc:824] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.43.125@KRBTEST.COM
I20260810 14:40:14.234419 14457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: No bootstrap required, opened a new log
I20260810 14:40:14.234874 14538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: No bootstrap required, opened a new log
I20260810 14:40:14.238480 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.225953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.21.43.125:36197)
0810 14:40:14.226187 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.226191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.226348 (+ 157us) server_negotiation.cc:407] Connection header received
0810 14:40:14.226542 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.226545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.226589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.226637 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.227770 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.228538 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.229633 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.229953 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.230036 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.233461 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.233465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.233468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.233495 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.235729 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.236490 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.236494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.236497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.236607 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.236948 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.236951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.236953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.237531 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.237741 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.238160 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.238305 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
Aug 10 14:40:14 dist-test-slave-chsk krb5kdc[14366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.43.124@KRBTEST.COM for kudu/127.21.43.124@KRBTEST.COM
I20260810 14:40:14.239586 14538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.239876 14538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed7fbb88b904957821772410ab8be8a, State: Initialized, Role: FOLLOWER
I20260810 14:40:14.239831 14457 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.240072 14457 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4854e16a894b11960b6483a1fb84f2, State: Initialized, Role: FOLLOWER
I20260810 14:40:14.240296 14538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.240415 14457 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.241955 14538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:14.241958 14457 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:14.242897 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.227204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.43.124:37439 (local address 127.0.0.1:58756)
0810 14:40:14.232810 (+ 5606us) negotiation.cc:107] Waiting for socket to connect
0810 14:40:14.232825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:40:14.232950 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:40:14.233434 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:40:14.233440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:40:14.233664 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:40:14.234104 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.234117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.235424 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.235427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.236315 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:40:14.236352 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.237009 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.237015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:40:14.237092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.237601 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:40:14.237616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:40:14.239214 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:40:14.241232 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.241235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.241237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.241466 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.241759 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:40:14.241762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:40:14.241764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:40:14.241838 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:40:14.242300 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:40:14.242303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:40:14.242401 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 14:40:14.242499 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 14:40:14.242670 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5518}
I20260810 14:40:14.243468 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.227265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.21.43.124:37439)
0810 14:40:14.227380 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.227386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.232900 (+ 5514us) server_negotiation.cc:407] Connection header received
0810 14:40:14.233062 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.233064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.233111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.233185 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.234249 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.235304 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.236477 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.236794 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.236888 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.239352 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.239355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.239357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.239382 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.241116 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.241583 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.241586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.241587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.241666 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.241921 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.241924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.241926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.242171 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.242264 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.243192 (+ 928us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.243317 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:14.243834 21677 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:40:14.245253 14427 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ee4854e16a894b11960b6483a1fb84f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51180
I20260810 14:40:14.245532 14427 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:40:14.245639 14427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:14.246771 14623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: Bootstrap starting.
I20260810 14:40:14.246975 14427 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.248014 14427 leader_election.cc:288] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed7fbb88b904957821772410ab8be8a (127.21.43.125:36197), 0884fba5cc9a4664b7d01d8c431d9e38 (127.21.43.124:37439)
I20260810 14:40:14.248517 14512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4854e16a894b11960b6483a1fb84f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ed7fbb88b904957821772410ab8be8a"
I20260810 14:40:14.248754 14512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:14.249537 14623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: Neither blocks nor log segments found. Creating new log.
I20260810 14:40:14.250232 14512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4854e16a894b11960b6483a1fb84f2 in term 1.
I20260810 14:40:14.250514 14623 log.cc:824] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: Log is configured to *not* fsync() on all Append() calls
I20260810 14:40:14.250710 14403 leader_election.cc:302] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed7fbb88b904957821772410ab8be8a, ee4854e16a894b11960b6483a1fb84f2; no voters:
I20260810 14:40:14.251127 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.251509 14636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:40:14.251786 14636 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [term 1 LEADER]: Becoming Leader. State: Replica: ee4854e16a894b11960b6483a1fb84f2, State: Running, Role: LEADER
I20260810 14:40:14.251909 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.252322 14636 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.252393 14668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.252611 14668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.254591 14623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: No bootstrap required, opened a new log
W20260810 14:40:14.257944 14661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 14:40:14.258966 14623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.259276 14623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0884fba5cc9a4664b7d01d8c431d9e38, State: Initialized, Role: FOLLOWER
I20260810 14:40:14.259639 14623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } }
I20260810 14:40:14.259722 14670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4854e16a894b11960b6483a1fb84f2. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.259837 14670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: This master's current role is: LEADER
W20260810 14:40:14.260787 14656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:14.260913 14656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 14:40:14.261052 14661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:14.261219 14623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:40:14.261313 14593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee4854e16a894b11960b6483a1fb84f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0884fba5cc9a4664b7d01d8c431d9e38"
I20260810 14:40:14.261541 14593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:40:14.263187 14593 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4854e16a894b11960b6483a1fb84f2 in term 1.
W20260810 14:40:14.267091 14691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:40:14.267210 14691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:40:14.269418 14465 master_runner.cc:263] No leader master found from master 3ed7fbb88b904957821772410ab8be8a
I20260810 14:40:14.269500 14465 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:40:14.270452 14678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:40:14.273192 14678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:40:14.274277 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.253036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.21.43.125:36197)
0810 14:40:14.253161 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.253178 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.253756 (+ 578us) server_negotiation.cc:407] Connection header received
0810 14:40:14.254218 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.254224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.254280 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.254336 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.256004 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.256769 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.258642 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.258963 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.259049 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.270191 (+ 11142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.270197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.270201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.270234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.272176 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.272709 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.272713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.272714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.272807 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.273220 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.273223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.273226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.273459 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.273595 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.273987 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.274126 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:40:14.278203 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.253777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.21.43.124:37439)
0810 14:40:14.253893 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.253899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.264530 (+ 10631us) server_negotiation.cc:407] Connection header received
0810 14:40:14.264683 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.264686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.264737 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.264811 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.267364 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.268162 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.269643 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.269981 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.270073 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.274344 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.274348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.274351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.274380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.276247 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.276789 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.276793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.276795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.276887 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.277206 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.277210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.277214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.277433 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.277547 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.277899 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.278025 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:40:14.279335 14676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.279470 14676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.284255 14512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [term 1 FOLLOWER]: Refusing update from remote peer ee4854e16a894b11960b6483a1fb84f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:14.284353 14593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [term 1 FOLLOWER]: Refusing update from remote peer ee4854e16a894b11960b6483a1fb84f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:40:14.285106 14636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:14.285655 14636 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:40:14.289809 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4854e16a894b11960b6483a1fb84f2. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.289973 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.290239 14676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee4854e16a894b11960b6483a1fb84f2. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.290373 14676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.292613 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:14.254623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.21.43.126:39803)
0810 14:40:14.254740 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.254745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.275809 (+ 21064us) server_negotiation.cc:407] Connection header received
0810 14:40:14.275999 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.276002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.276057 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.276123 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.276989 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.277781 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.280031 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:14.280389 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:14.280473 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:14.282428 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:40:14.282432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:40:14.282435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:40:14.282466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:40:14.285755 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.290618 (+ 4863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.290622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.290624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.290755 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:40:14.291083 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:40:14.291086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:40:14.291088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:40:14.291365 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:40:14.291511 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:14.292310 (+ 799us) server_negotiation.cc:299] Negotiation successful
0810 14:40:14.292446 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:40:14.298385 14668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.298547 14668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:14.298873 14693 mvcc.cc:205] Tried to move back new op lower bound from 7316983047299653632 to 7316983047182426112. Current Snapshot: MvccSnapshot[applied={T|T < 7316983047299653632}]
I20260810 14:40:14.301414 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.301543 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.302147 14695 mvcc.cc:205] Tried to move back new op lower bound from 7316983047299653632 to 7316983047182426112. Current Snapshot: MvccSnapshot[applied={T|T < 7316983047299653632}]
I20260810 14:40:14.302805 14670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.302932 14670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:40:14.304116 14676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee4854e16a894b11960b6483a1fb84f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4854e16a894b11960b6483a1fb84f2" member_type: VOTER last_known_addr { host: "127.21.43.126" port: 39803 } } peers { permanent_uuid: "3ed7fbb88b904957821772410ab8be8a" member_type: VOTER last_known_addr { host: "127.21.43.125" port: 36197 } } peers { permanent_uuid: "0884fba5cc9a4664b7d01d8c431d9e38" member_type: VOTER last_known_addr { host: "127.21.43.124" port: 37439 } } }
I20260810 14:40:14.304246 14676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:40:14.304594 14678 catalog_manager.cc:1347] Generated new cluster ID: 28b1ab243bac4520a0bbc0f110cf473d
I20260810 14:40:14.304662 14678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:40:14.304908 14694 mvcc.cc:205] Tried to move back new op lower bound from 7316983047299653632 to 7316983047182426112. Current Snapshot: MvccSnapshot[applied={T|T < 7316983047299653632}]
I20260810 14:40:14.334108 14678 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:40:14.335485 14678 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:40:14.352396 14678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee4854e16a894b11960b6483a1fb84f2: Generated new TSK 0
I20260810 14:40:14.353397 14678 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:40:14.395473 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.392755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.21.43.125:36197)
0810 14:40:14.392891 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.392896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.394260 (+ 1364us) server_negotiation.cc:407] Connection header received
0810 14:40:14.394480 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.394484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.394538 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.394606 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.395324 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34026: BlockingRecv error: recv got EOF from 127.0.0.1:34026 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 14:40:14.396999 14626 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.393631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.21.43.124:37439)
0810 14:40:14.393752 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.393756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.395992 (+ 2236us) server_negotiation.cc:407] Connection header received
0810 14:40:14.396173 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.396176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.396229 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.396282 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.396884 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58780: BlockingRecv error: recv got EOF from 127.0.0.1:58780 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 14:40:14.399013 14461 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.397520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.21.43.126:39803)
0810 14:40:14.397651 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.397655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.398090 (+ 435us) server_negotiation.cc:407] Connection header received
0810 14:40:14.398261 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.398264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.398314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.398366 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.398890 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51188: BlockingRecv error: recv got EOF from 127.0.0.1:51188 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 14:40:14.403692 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.401953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.21.43.125:36197)
0810 14:40:14.402095 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.402099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.402614 (+ 515us) server_negotiation.cc:407] Connection header received
0810 14:40:14.402865 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.402869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.402951 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.403005 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.403571 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34036: BlockingRecv error: recv got EOF from 127.0.0.1:34036 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 14:40:14.408778 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.407190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.21.43.125:36197)
0810 14:40:14.407301 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.407305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.407850 (+ 545us) server_negotiation.cc:407] Connection header received
0810 14:40:14.408045 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.408048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.408098 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.408152 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.408669 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34044: BlockingRecv error: recv got EOF from 127.0.0.1:34044 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 14:40:14.414315 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.412770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.21.43.125:36197)
0810 14:40:14.412874 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.412878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.413407 (+ 529us) server_negotiation.cc:407] Connection header received
0810 14:40:14.413606 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.413609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.413659 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.413710 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.414209 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34058: BlockingRecv error: recv got EOF from 127.0.0.1:34058 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:40:14.422132 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.420429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.21.43.125:36197)
0810 14:40:14.420570 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.420574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.421128 (+ 554us) server_negotiation.cc:407] Connection header received
0810 14:40:14.421332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.421335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.421389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.421443 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.422010 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34072: BlockingRecv error: recv got EOF from 127.0.0.1:34072 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 14:40:14.435211 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.433197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.21.43.125:36197)
0810 14:40:14.433326 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.433331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.434037 (+ 706us) server_negotiation.cc:407] Connection header received
0810 14:40:14.434267 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.434271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.434328 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.434385 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.435052 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34082: BlockingRecv error: recv got EOF from 127.0.0.1:34082 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 14:40:14.456192 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.454073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.21.43.125:36197)
0810 14:40:14.454231 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.454236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.454929 (+ 693us) server_negotiation.cc:407] Connection header received
0810 14:40:14.455227 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.455231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.455295 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.455400 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.456047 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34088: BlockingRecv error: recv got EOF from 127.0.0.1:34088 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 14:40:14.492144 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.490748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.21.43.125:36197)
0810 14:40:14.490912 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.490916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.491174 (+ 258us) server_negotiation.cc:407] Connection header received
0810 14:40:14.491349 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.491353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.491413 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.491503 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.491989 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34102: BlockingRecv error: recv got EOF from 127.0.0.1:34102 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 14:40:14.559927 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.558511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.21.43.125:36197)
0810 14:40:14.558699 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.558703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.558725 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:40:14.558924 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.558927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.558981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.559066 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.559767 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34112: BlockingRecv error: recv got EOF from 127.0.0.1:34112 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 14:40:14.691772 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.690441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.21.43.125:36197)
0810 14:40:14.690615 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.690620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.690696 (+ 76us) server_negotiation.cc:407] Connection header received
0810 14:40:14.690825 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.690827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.690880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.690969 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.691634 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 14:40:14.951288 14545 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:40:14.949985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.21.43.125:36197)
0810 14:40:14.950137 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:14.950144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:14.950265 (+ 121us) server_negotiation.cc:407] Connection header received
0810 14:40:14.950447 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:14.950450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:14.950510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:14.950603 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:40:14.951133 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34116: BlockingRecv error: recv got EOF from 127.0.0.1:34116 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:40:15.262861 14661 catalog_manager.cc:1259] Loaded cluster ID: 28b1ab243bac4520a0bbc0f110cf473d
I20260810 14:40:15.262955 14661 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: loading cluster ID for follower catalog manager: success
I20260810 14:40:15.264829 14661 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: acquiring CA information for follower catalog manager: success
I20260810 14:40:15.266112 14661 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ed7fbb88b904957821772410ab8be8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:40:15.268504 14691 catalog_manager.cc:1259] Loaded cluster ID: 28b1ab243bac4520a0bbc0f110cf473d
I20260810 14:40:15.268565 14691 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: loading cluster ID for follower catalog manager: success
I20260810 14:40:15.270185 14691 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: acquiring CA information for follower catalog manager: success
I20260810 14:40:15.271399 14691 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0884fba5cc9a4664b7d01d8c431d9e38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:40:15.469415 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:15.465539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.21.43.125:36197)
0810 14:40:15.465822 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:15.465826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:15.465843 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:15.465979 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:15.465981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:15.466119 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:15.466223 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:15.466473 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.466995 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.467889 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.468125 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.468210 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:15.468535 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:15.468980 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:15.469077 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:15.469144 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:40:15.469200 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260810 14:40:15.476599 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:15.472709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.21.43.124:37439)
0810 14:40:15.472993 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:15.473004 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:15.473018 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:40:15.473103 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:15.473106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:15.473223 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:15.473324 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:15.473517 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.474034 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.474878 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.475133 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.475298 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:15.475636 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:15.476100 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:15.476207 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:15.476292 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 14:40:15.476346 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260810 14:40:15.484139 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:15.480260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.21.43.125:36197)
0810 14:40:15.480545 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:15.480549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:15.480566 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:40:15.480626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:15.480629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:15.480673 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:15.480741 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:15.481034 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.481836 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.482914 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.483207 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.483321 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:15.483608 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:15.483695 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:15.483759 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:15.483899 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 14:40:15.483959 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260810 14:40:15.485102 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:15.480380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.21.43.124:37439)
0810 14:40:15.480513 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:15.480518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:15.481035 (+ 517us) server_negotiation.cc:407] Connection header received
0810 14:40:15.481184 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:15.481189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:15.481245 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:15.481310 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:15.481607 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.482268 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.483661 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.483905 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.484033 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:15.484647 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:15.484732 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:15.484808 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:15.484900 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 14:40:15.484951 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:40:15.485472 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:40:15.480212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.21.43.126:39803)
0810 14:40:15.480767 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0810 14:40:15.480772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:40:15.480785 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:40:15.480881 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:40:15.480884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:40:15.481098 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:40:15.481240 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:40:15.481459 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.482168 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.483626 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:40:15.483983 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:40:15.484088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:40:15.484514 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:40:15.484855 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:40:15.484945 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:40:15.485259 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 14:40:15.485327 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":84,"threads_started":1}
I20260810 14:40:15.494210 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 14381
I20260810 14:40:15.504141 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 14465
I20260810 14:40:15.512238 21677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA76VjH/build/debug/bin/kudu with pid 14546
2026-08-10T14:40:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3186 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6358 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (297729 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST